Class 18 – Crawl Budget Complete Guide |
FREE Advanced SEO Course With Notes
Watch the Full Video : Crawl Budget Complete Guide
Google decides for each individual website how many resources it should spend on that website, how many pages it should crawl, etc. This process is called the crawl budget. Like, suppose your website has 100000 pages, then Google will decide how many pages out of those 100000 pages it should crawl and how many pages it should not crawl.
If a website has 10000 pages, then there is no need to worry about the crawl budget because websites with fewer than 10000 pages do not face crawl budget issues.
The crawl budget depends on two things
One is Resources, and the other is Value.
Value:
- How unique is the content of your website?
- How frequently are you posting content on your website?
- How established is your website (in your niche)?
Resources/Tech
TTFB = Time To First Byte: When Google’s servers or Googlebot send a request to your website to access a particular file or visit a particular URL, how long your server takes to start attending that request, that time taken to access the file or URL tells Google how responsive your server is.
Now, to know whether your server is slow or fast based on TTFB, Google has a tool called https://pagespeed.web.dev/. On this tool, you can add your website URL and click on Analyze, then you will know what the TTFB time is.
Average Response Time (in GSC): To check the Average Response Time, you have to go to Google Search Console, then go to Settings, and there you have to click on “Crawl Stats”, and you will see the Average Response Time there. This time is shown by combining all the URLs. According to Google, this time should be less than 200 milliseconds, but even if it is between 200 and 500 milliseconds, there is no problem, but it should not be more than that.
If you want to know where Googlebot’s time is being spent, then scroll down the “Crawl Stats” page in Google Search Console. There you will find the “By response” section, and from there you will know everything about where the time is being spent. “By response” means what Googlebot is visiting the most every time it comes to your website.
OK (200) means Googlebot sent a request to the server, and the server responded properly and provided the information. It means everything is fine.
Not Found (404) means Googlebot sent a request, but the server could not respond to it because the request is incorrect. 400-series errors do not impact Google’s crawl budget.
404 does not impact the crawl budget, but it also does not provide complete information. That is why you should use 410, which means the page is gone. To implement 410, you will need the help of a developer.
https://httpstatus.io/ is a tool that you can use to see the complete redirection chain.
Errors that start with 500 impact Google’s crawl budget. Like, Googlebot sent a request to your website to access a file or URL, and the file or URL exists on your website, but your server is still not able to provide the information to Googlebot, which means there are issues with your server.
OK (200) means the pages are being crawled properly, but that does not mean you should not check them. You should click on it and see which pages Googlebot is actually crawling. If it is crawling pages with UTM parameters, “fbclid”, or similar types of pages again and again, then you should disallow them from being crawled.
If you want to disallow these pages, then you have to create a rule in the robots.txt file. Copy the URL and create the rule in the robots.txt file.
User-agent: Googlebot
Disallow: /?utm_source”influencer_program