If the buffer contains a chemical it is probably a good idea to take the fish out, also to wash the tank really well before setting it up again. Fish can be sensitive to chemicals used for cleaning, in handcreams, insect killers etc. Try to keep all chemicals away from the tank.
I take it you are talking about polishing the outside of the tank. If you are talking about the inside, you would absolutely have to take fish and water out.