Can't speak for all the removers out there, but prime dissipates in just about 24 hours with or without chlorine. It won't hurt the fish at all to use it, just doesn't really do anything for them either.
If you are on well water it shouldn't be needed, but you can't know for sure unless you have the water tested. My county requires a years test for pathogens, but for a few extra bucks I can have it expanded to a full breakdown.
If you are on city water there is a very good chance you have at least trace amounts of chlorine in it. I know there are some cities in southern Colorado that just pump straight from the wells into the city pipes, but it is pretty rare. Either way a call to the water company can settle it for you.