I leave them. I like to think of them as trophies for the fish. Maybe they keep score and if I start removing them it'll throw off the count and ruin the contest.
I had one of my corys get stuck in a mystery snail shell. I had to hold it under the surface and break off pieces until I could get him out. He was still alive but I have no idea how long he had been in there. Now I take out the larger shells.