Being visibly dirty has absolutely nothing to do with the quantity of harmful toxins in the water, cleaning the sides of the tank does not remove toxins from the water, and depending on how you're "cleaning the filter", you could be removing beneficial bacteria needed to process fish waste properly. We've explained the most likely reason that this happened to your tank. You might go back over some of the replies to your post, and google the things they mention. It's important to understand how the relationship between the nitrogen cycle, ammonia, and PH all effect each other in home aquaria. It sounds like you don't quite grasp what's been suggested as the cause.
I'm glad to hear you've been lucky this far, but overstocking a tank, not cleaning it often, and then doing 100% water changes when you do, is hard on any fish, and can cause serious problems (including death), which is more than likely exactly what you are experiencing now. I understand that you haven't had problems with this in the past, but not to sound like a jerk, if your 6 goldfish "coped fine" in the 20 gallon tank with your maintenance routine, chances are you'd still have 6 of them.
For most goldfish, 20 gallons is too small for one adult, let alone 3, or 6 (granted they may be small now). The best thing you can do is either look for a larger tank, or find them a home with someone that has an appropriate sized tank to home them in. If that's doesn't work for you, the next best thing you can do is make a serious effort to keep up on your water changes, and do them properly, for the safety of the fish. Do them more often (for goldfish at least weekly), and change less water (no more than 50%) each time.
One more consideration, you said the temperature was matched... but what was the temperature of the water? I know my tanks are all running in the high 70's, low 80's right now, and goldfish are a coldwater fish. I'm no goldfish expert, but it's another possibility, though my money is on the maintenance routines.