Depending on the size of your tank, it is either simply an issue of aesthetics or serious water quality. I have a 72g tank, I leave some of the little dead/decayed Amazon sword leaves in. I don't get much dead other than that. Floaters (plants, not fish) I leave in if they are green, brown they go.