Water changes can be stressful if there are excessive changes in water chemistry or temperature.
More $$$$ spend on Prime or whatever you use.
Greater possibility of your fish being poisoned by some oddball chemical that was released by accident in your water supply. This does occasionally happen.
Everything else being equal, your fish would prefer that you muck around in their tank as little as possible.
But all in all, too many is certainly better than too few.