Crawling and indexing have been a hot topic over the last few years. As soon as Google launched Google Panda, people rushed to their server logs and crawling stats and began fixing their index bloat. All those problems didn’t exist in the “SEO = backlinks” era from a few years ago. With this exponential growth of technical SEO, we need to get more and more technical. That being said, we still don’t know how exactly Google crawls our websites. Many SEO still can’t tell the difference between crawling and indexing.
The biggest problem, though, is that when we want to troubleshoot indexing problems, the only tool in our arsenal is Google Search Console and the Fetch and Render tool. Once your website includes more than HTML and CSS, there’s a lot of guesswork into how your content will be indexed by Google. This approach is risky, expensive, and can fail multiple times. Even when you discover the pieces of your website that weren’t indexed properly, it’s extremely difficult to get to the bottom of the problem and find the fragments of code responsible for the indexing problems.
So what does Ilya’s revelation in this tweet mean for SEOs?
Knowing that Chrome 41 is the technology behind the Web Rendering Service is a game-changer. Before this announcement, our only solution was to use Fetch and Render in Google Search Console to see our page rendered by the Website Rendering Service (WRS). This means we can troubleshoot technical problems that would otherwise have required experimenting and creating staging environments. Now, all you need to do is download and install Chrome 41 to see how your website loads in the browser.
Google uses Chrome 41 for rendering. What does that mean?
Fetch and Render is the Chrome v. 41 preview
There’s another interesting thing about Chrome 41. Google Search Console’s Fetch and Render tool is simply the Chrome 41 preview. The righthand-side view (“This is how a visitor to your website would have seen the page”) is generated by the Google Search Console bot
Chrome 41 vs Fetch as Google: A word of caution
Chrome 41 is a great tool for debugging Googlebot. However, sometimes (not often) there’s a situation in which Chrome 41 renders a page properly, but the screenshots from Google Fetch and Render suggest that Google can’t handle the page. It could be caused by CSS animations and transitions, Googlebot timeouts, or the usage of features that Googlebot doesn’t support.
What features are supported by Googlebot and WRS?
According to the Rendering of Google Search guide:
- Googlebot doesn’t support IndexedDB, WebSQL, and WebGL.
- HTTP cookies and local storage, as well as session storage, are cleared between page loads.
- All features requiring user permissions (like Notifications API, clipboard, push, device-info) are disabled.
- Google can’t index 3D and VR content.
- Googlebot only supports HTTP/1.1 crawling.
No HTTP/2 support (still)
We’ve mostly been covering how Googlebot uses Chrome, but there’s another recent discovery to keep in mind.
Rumor has it that Chrome 59’s headless mode was created for Googlebot, or at least that it was discussed during the design process. It’s hard to say if any of this chatter is true, but if it is, it means that to some extent, Googlebot will “see” the website in the same way as regular Internet users.
Chrome 41 vs. Googlebot’s crawling efficiency