Did you boil them before putting them in your tank? If not, then the problem is most likely a fungal growth. Yes, it will hurt your fish. I'm pretty sure it will only hurt fish that touch or ingest the fungus, though. My mother put a piece of driftwood in her tank, the same thing happened, and the pleco that was eating the fungus shortly died of a fungal infection....
Take the driftwood out of your tank (carefully, so as not to rub the stuff off into your tank) and pour lots of boiling water over it, then leave it outside the tank to dry. Then do it again. Let it sit for a few days in a fairly dry environment, like near a heater vent or on your fireplace hearth, if you're lucky enough to have one, to air out before putting it back in the tank. In the meantime, treat your tank for fungus if there is any growing anywhere else in the tank. You might also want to do a good sized water change to dilute any particles that have gotten into the water.