I would remove the fish, so if any anerobic pockets were opened up while scooping out the old media, the fish won't suffer. Then I would use that opportunity to do a good water change (30%) I don't know if I'm right or wrong, but that is what I do.
