I wouldn't leave them off for that long.Nice thinking, so i can do my water change straight from the tap just leave the filter off for a couple of hours while the water settles and all is fine.
I switch my filters off, drain out some water and gravel vac, fill it back up adding dechlor in the process, then turn the pumps back on. The whole process might take 30 minutes if I am doing a big cleaning/pruning but is even faster if I am just changing out water.
By the way- I am blessed with very warm groundwater so temp fluctuation is not an issue for me.