See this page with Table Tennis rules. Specifically there are two rules to be read here.
First, a legal serve is defined as:
2.06.03 As the ball is falling the server shall strike it so that it touches first his or her court and then, after passing over or around the net assembly, touches directly the receiver's court; in doubles, the ball shall touch successively the right half court of server and receiver.
Keeping that in mind, your scenario of a serve touching the net, and then landing back in the server's court would not be a let. Reason being, a let is essentially a legal serve if you remove the touch of the net from the equation. i.e.:
If the serve is legal except that it touches the net, it is called a let serve. Let serves are not scored and are reserved.
The above remains correct after checking the most credible source I could find, but FYI here's the link to what I believe are the official rules: http://www.ittf.com/ittf_handbook/ittf_hb.html -- Check the link on the left for A Let.