Well, first off, make sure you NEVER wash the media with tap water, always with tank water or dechlorinated water, tap kills the BB. (You didn't specify if you did or not, just thought I'd let you know :] )
It's amazing my fish lived so long -- I have 5 fish that had lived in their original tank for 7 years before getting their new digs and I had been rinsing the sponge and the filter with tap water

My newly established tank doesn't actually have a sponge, it's a bio wheel type -- :idea: but hey, it's got room for another "bio bag" - could I plunk into that space the sponge from the whisper filter and leave it for a few days and then put the sponge into the new tank???