I wouldn't. You need to have a step or grade of mechanical filtration for your filter to work properly (unless you plan on cleaning it very often). The incoming water should pass through media with larger bypass first, followed by smaller bypass, and pass through the finest material last. This allows the bigger "gunk" to get trapped first that would otherwise plug up the next in line media (the finer media). None of your media is going to work well if the first media is plugged and restricting water flow!
If you really aren't getting much dirt out of your black sponges, replace the more porous ones (20 pore per inch) with white floss but keep the finer black sponge (30 pores/inch) in line before the floss. Good luck!