Nitrifying bacteria adhere to objects. They can be removed, but it takes a bit more than some sloshing and suction from a gravel vac. Similar for rinsing bacteria, but if you rinse it in chlorinated tapwater, the chlorine can kill the bacteria, so should be done with caution, or in cases where the sponge being rinsed is not the primary biological filtration.