What’s the difference between robots.txt and Meta Tags?

There are several types of Meta Tags, including Title, Description, Keywords. You place Meta Tags in the “head” section of your Web pages HTML to provide information that helps control robots and crawlers searching your website. The information in Meta Tags is not viewable by site visitors unless they view the page’s source.

A robots.txt file tells search engine crawlers which pages or files the crawler can or can’t request from your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, you should use noindex directives, or password-protect your page. A robots.txt file specifies which parts of your Web page robots or crawlers can access. While some can ignore your robots.txt file, many search engines will find it and follow the specified protocol. You create and place a robots.txt file in the top-level directory of your Web server.

While both a robots.txt file and a Meta Tag communicate the preferences of your website to search engines attempting to crawl and collect information, using a robots.txt file is recommended. The robots.txt file allows for more flexibility and control over what gets searched. It should be uploaded to the hosted site’s root directory.