won't hurt them a bit.
when you get more substrate, give it a good rinse to ensure it is free from silt and particulate that could otherwise get into your filter and wreck the impeller inside it. then just add it to the tank with a measuring cup or other container, slowly lowering it down to the tank bottom, then spread it out where you want it. no need to remove fish at all for this. i've done it many times, with both gravel and sand.