It depends on the size and species. Some of the large crayfish really lack power in their claws (my large Cherax quads for example. It gets your attention by it doesn't really hurt). And some smaller crayfish are capable of putting a heck of a lot of power on single points, and can break the skin (my 4" Cherax holthuisi has hooked claws, so when she pinches all the power is focused on the hooked points, hurts like all hell).
For the most part though, it just gets your attention but it's not going to make anyone run off crying or anything. And with the exception of a few species, if you pick them up from behind the head, they can't get their claws back far enough to pinch you. ^_^