The robots tag tells search engine spiders to index your page and follow all links, or not. The options are;
Webmasters should also consider including a robots.txt file in the root directory of their site. Most search engine spiders will request this file, and follow instructions on which files/sub-directories to ignore and therefore not index.
Even if your site has no need for a robots.txt file, it is often still good practice to provide a blank file so that the search engines don't index your error page.