It shouldn’t harm the skimmer. If anything the grit may shorten the life of the needle wheel pump used to pump water through the skimmer. Didn’t know you were using filter wool in the outtake box. That will indeed help with the cloudiness.
However, during the cycle phase of a new tank, you don’t want to remove the things that are going to create ammonia. Ammonia needs to be present so the nitrifying bacteria, that eats ammonia, will grow and multiply. Using the skimmer prior to having the cycle complete would hamper the process to some degree.