The dirt smell is created by Mycobacterium sp. of bacteria. These are found almost everywhere in the soil and water. They are a major contributor to breaking down organic waste. I've actually grown this species in a test tube. When you pop the cap off it smells like you've just stuck you head in some dirt.
They are part of the normal flora of your tank and usually means that it is healthy. Carbon is the only thing I've found that can remove the odors produced by it from the water column.