Class 17 – Robots.txt Complete Guide |
FREE Advanced SEO Course With Notes
Watch the Full Video : Robots.txt Complete Guide
The robots.txt file is so powerful that it can affect the crawling, indexing, and ranking of any website. robots.txt is a very simple file in which rules are written in a very simple format, but just because it is simple, people take it lightly and do not understand its real power.
There are many rules for robots.txt that are not applicable to other files, such as its name!
robots.txt file must be placed in the root folder of the website, means it has to be placed in the very first folder of the website.
Exa: mitromate.com/robots.txt
The location of the robots.txt file is fixed because no one tells any bot the location of the robots.txt file. Whenever any bot crawls your website, it reads the robots.txt file first.
User-agent: *
Disallow: /wp-json/
Disallow: /?rest_route=
Allow: /
Sitemap: https://mitromate.com/sitemap_index.xml
User-agent: Googlebot – It means that you are giving only Googlebot permission to crawl your website, and you are not allowing the bots of any other search engine.
User-agent: * – It means that you are allowing any search engine to crawl your website.
Disallow means that you are not giving permission. Allow means that you are giving permission.
Disallow: / -It means that you have disallowed all the files, all the URLs, the entire website, and the entire web host because every URL of any website starts with a forward slash (/), which is the root directory URL. So whenever you add a / in the robots.txt file, it means that you have disallowed everything. No bot can access any page or any file.
Allow: / – It means that you have given all bots permission to crawl your website. Any search engine bot can crawl your website.
If you use both Allow and Disallow rules together, then Google or any search engine bot will follow the Allow rule because no search engine or its bots want you to keep your website’s information hidden. Even if we apply the Disallow rule 10 times and the Allow rule only once, search engine bots will still follow the Allow rule.
One mistake that many SEOs make is that if there is a page they do not want to keep indexed, they add the noindex tag to that page and also disallow the same page in the robots.txt file. As you know, search engine bots read the website’s robots.txt file first. So if you have disallowed that page, the bots will not visit it because you have not given permission. In this situation, your page will remain indexed even though you have added the noindex tag because you have disallowed it in the robots.txt file. Search engine bots will not visit the page, and if they do not visit it, how will they know that you have added the noindex tag to the page?
If you want search engine bots not to crawl or index all the articles and pages inside a particular category of your website, then you have to add a Disallow rule in the robots.txt file. (Example: Disallow: /actors-biography/*) By applying this rule, search engine bots will not visit any of the articles in that category of your website.
You can create different rules for different search engine bots, like:
User-agent: Googlebot
Disallow: /paid-articles/*
User-agent: Bingbot
Disallow: /
You should also add your website’s sitemap to the robots.txt file because the process of discovering a website starts with the robots.txt file.
Google has always given priority to the robots.txt file, but now it gives it even more priority. That is why Google has added a robots.txt section in Google Search Console. You can go to Settings and check the status of your robots.txt file, like when it was last crawled, whether it was fetched or not, etc.
Suppose that by mistake, one of your employees deletes your website’s robots.txt file, and Google visits your robots.txt file and finds that it does not exist. In that case, after noticing it, Google will not crawl your website or any page of your website for 12 hours. After 12 hours, if Google still cannot find your robots.txt file, it will use the old version of your website’s robots.txt file that it has stored and follow the rules in it for the next 30 days. If Google still cannot find your website’s robots.txt file after 30 days, then Google will assume that your website does not have a robots.txt file and will crawl your entire website without any rules.
https://technicalseo.com/ is a website that you can use to check the technical SEO part of your website. It is a very useful tool.