I use the same rules as if I was going to touch food. If I know my hands are fairly clean then I just dive in. If not I wash with soap. I am really picky about soap residue so I will either wait an hour or rinse my hands in soda water as the acid rinses soap faster. (It's an OCD thing I know.)
Once I am done I always wash in soapy water.
Once I am done I always wash in soapy water.