Short answer, they do indeed sweep the track, as you can see in the image shown.
This may not happen just before the race however, as most race weekends have a lot of other races happening, support races, so it isn't always possible to sweep the track just before the F1 race begins. As you can imagine it is quite a time-consuming process to clean the track.
You'll often hear comentators talking about the track being "green" and therefore slippery, this isn't the same as the track being dirty. Motor racing tracks have a certain "line" that the cars take to get maximum speed, this racing line gets more and more rubber laid down on it as cars pass by, the rubber gives more grip and so cars will go quicker. If a track isn't used for a while, this rubber is washed away by the rain and generally by weather effects, so the track becomes greener.
Some tracks that are only used for races occasionally, ie Montreal, will be very green at the start of the race weekend but "rubber in" over the course of the weekend.
You'll notice that as well as rubbering in the racing line, modern racing tyres produce something called "marbles" that are in fact small pieces of rubber than fall off the tyres. These often lie off the racing line where cars don't usually go and so don't get "swept up" by the passing traffic. These can become very slippery if you go off line and run over them if they accumulate in sufficient number.
Some tracks, ie Bahrain are prone to dust and sand blowing in etc. You'll sometimes find if there's been an oil spill on track, they put cement dust on it and then use a road sweeper to clean it up. They sometimes do this during a race if it's been red flagged etc and for small pieces of debris occasionally.
Just Google "road sweeper F1 track" for more images of this type of thing.