Well, depends on what you're doing and the likelihood of squashdom. Personally I do leave mine in but they are fairly hardy and I just go carefully when shifting the stuff around.
This was basically my thought process. Some people don't like to move things around while their fish are still in the tank. It's just as stressful leaving them in as it is taking them out IMO so it really comes down to how comfortable the OP is with it.