Definitely keep an eye on your inhabitants and your live rock. It is probably knocking out holes in your LR to make its cave. A mantis will eventually reveal himself, a pistol will very rarely if ever come out. But Grins is right, if it is a pistol your cleaner shrimp is in trouble.
I have a pistol now, at first I thought it was a mantis, but then I found him and fed him, could tell right away it wasn't a mantis. I also found the graveyard! Right behind his burrow there is a massive pile of hermit shells, that is how he was feeding, he was eating all my hermits!
In my case I didn't know that I had him and I could never figure out why my hermits would disappear so fast. Anyways, now I feed him frozen silver sides 3 times a week and my hermits can live longer lives.