Depends. Some fishing sinkers are made from lead, which is toxic to pretty much everything; however, many newer sinkers are made from various other metals, such as tungsten, which are pretty inert.
*EDIT* The little round 'split shot' sinkers are almost always lead based. I recommend removing them immediately. Try stainless steel washers or other hardware instead.