Google’s John Mueller was questioned in Google’s latest Q&A video on concerns of their site not being crawled often enough. Here is the question below:
“How often does Google re-index a website? It seems like it’s much less often than it used to be. We add or remove pages from our site, and it’s weeks before those changes are reflected in Google Search.”
In the video, Mueller explains that Googlebot takes some time to crawl the entire website, and if there were a way to crawl a whole site in a short amount of time, it would cause unnecessary interference to the server. Therefore, Googlebot has a strict limit on how many pages it can crawl every day. How many? It was not mentioned.
John Mueller’s full response is quoted below.
“Looking at the whole website all at once, or even within a short period of time, can cause a significant load on a website. Googlebot tries to be polite and is limited to a certain number of pages every day. This number is automatically adjusted as we better recognize the limits of a website. Looking at portions of a website means that we have to prioritize how we crawl.
So how does this work? In general, Googlebot tries to crawl important pages more frequently to make sure that most critical pages are covered. Often this will be a websites homepage or maybe higher-level category pages. New content is often mentioned and linked from there, so it’s a great place for us to start. We’ll re-crawl these pages frequently, maybe every few days. maybe even much more frequently depending on the website.”
Now we know Googlebot is limited to the number of pages it can crawl. This tells us the most “important” pages to be crawled more frequently are the homepage and high-level category pages. Given this information, we understand that patience is the key when expecting changes on other pages from Google Search. However, one thing we do know is consistently posting relevant blogs/content every month will indeed bring in more value to your search results faster than let’s say, an old blog post. There are other factors on how to boost your old content for better SEO; please check out our blogs for more information.
If you are looking for an SEO company to boost your search results, we have you covered! Contact us for a free estimate at 857-526-0096 today!