What kind of filter do you have? When you say sponge "in there," I'm wondering if you mean the spongy plastic material designed to harbor the nitrifying bacteria like in a Tetra Whisper power filter. If so, water passes through it after being filtered by the filter cartridge, and it shouldn't collect very crud unless the cartridge is too clogged and causing water to flow over the cartridge instead of through it. Either way, you want to be careful cleaning it because you don't want to destroy the bacteria on it. At most, I'd swish it in the aquarium water. Chlorine or chloramines in tap water might be harmful.
Carbon is not normally washed, except to rinse it before its first use so not to cloud the water. As stated, it only lasts a few weeks anyway and is replaced. If you're asking about the filter cartridge that contains the carbon, I usually just swish it in my drained aquarium water with my weekly water change, and replace it altogether monthly.