What is Googlebot and how does it affect your website?

By 08/07/2019Jul 10th, 2019Development, SEO
Googlebot logo with bot

What is Googlebot?

Googlebot, (also called Google web crawler and Google spider) is a bot that crawls all of the web pages on the internet. It performs the same task as you would do if you went to a website and read through each and every page on it. But Googlebot does this for all websites. It performs three main tasks while on your website:


Crawling a website simply means looking at every page of that website, reading all of the information, viewing all of the images and videos that are there. Googlebot does this by looking first at the HTML of your site to find out what it is all about. It will also then follow any links from each page their pages and then crawl those too. Basically Google is trying to find out what each page is about so it can move on to indexing those pages.


Indexing web pages is very similar to how a library works with the Dewey Decimal System. A library indexes all of it’s books according to a system to make it easier for people to find the books they are looking for. Google does this with all your website pages so that it can make sense of it and provide the most relevant search results for people using search engines.

Informs PageRank

PageRank is the rating system Google gives to each web page on the internet. Google has over 200 signals that it uses to rank pages, which it does not disclose. Once Google has crawled your website it sends the information gathered to PageRank so that it can make an informed decision about how best to rank each page. Find out more about how to improve your PageRank.

How does Googlebot affect your website?

Google is very careful not to overload your website by over-crawling it as this could cause server errors or bandwidth problems. While crawling it first reads all of the HTML and then activates the Web Renderring Service (WRS). This makes it possible for Google to see your website as you intended it to be seen, by activating JavaScript (code that helps make your website look good and work well). Including too much JavaScript can slow this process down so using it only when necessary is a good idea. A mobile first approach is also a good idea to help Googlebot along it’s way. Google now focuses heavily on ranking websites that are fast and responsive to mobile devices.

But as Martin Splitt (WebMaster Trends Analyst, Google) says, “Just build good content for the users and you’ll be fine!”

If you would like to learn more about how to optimise your website for Googlebot, contact us today.

Sean McMahon

Author Sean McMahon

Sean McMahon is a digital strategist and website designer based in Johannesburg, South Africa. He has been in the digital design industry since 1998 and currently heads up Electric Pencil.

More posts by Sean McMahon

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.