Definitely go with the 12 for geese and swans. I agree with everything that has been said already on this thread. If you can, keep the 20 AND get the 12. I have seen guys take down geese and even swans with 20's, but I feel you greatly improve your chances by going with a 12.
I just picked up a 20 to use for the smaller birds (as suggested previously) and for the clay birds as well. It is easier on the shoulder and throws less lead out there for the clays. The way I figure it, with fewer pellets in the target area, you need to be a little better shot. If I can learn to be a better shot with the 20, it will also translate into shooting the 12 better. Only mistake I made is the two guns are not of the same make. I guess I'll just have to see if I can find a good deal on a 20ga BPS to keep more of the feel of the 12.
Is it even possible to ever have too many guns? Having too many guns is like having way too much time to devote to hunting. Impossible!