Jérôme says: “My tip is that you can use your server logs to run a state of indexation audit at scale, thanks to the ‘130 days’ rule discovered by fellow SEO Alexis Rylko.

We know that, if Googlebot hasn't visited a page in over 130 days, it's out of Google's index. This discovery was later studied further by fellow SEO Adam Gent, who is doing a lot of research on Google's indexation process as well.

You know that there is a report in Google Search Console, but with limitations. For example, the URL inspection API has strict daily query limits (2,000 queries per day), and the indexation report available in Search Console doesn't let you export a complete list of indexed or non-indexed URLs. So, that's a great way to bypass this limitation.”

Okay, so let's explore some of what you said in different parts.

You started off by talking about indexation audits. What is an indexation audit?

“Great question. It’s checking whether the pages on your website are indexed by Google or not.

Google is crawling the web using Googlebot, calling pages – and not all of them make it to the index. If you want to appear in the SERP, in the search results, you need to be indexed first. That's the first blocker to SEO. It's the first step that you need to unlock to make sure your content can be accessible/visible.”

Why would you really want to conduct one? Would you be nervous that you're not being indexed?

“Yes, because that's like working on the best content, that is reaching your target, dedicated to your persona, targeting the best keywords – everything is done perfectly – but Google does not have your page in its index, so it will never make it to the first page. You will never get clicks or impressions.

It's like having a house, but not the keys to get in.”

What do you typically find when analysing an indexation audit, and what do you do with the results?

“You're working on SEO on a website. It has many pages. Some of them are good for SEO. Others are not good enough, or not dedicated to SEO anyway. Google wants to find the best pages to answer users' questions.

It can be hard for Google to find your key pages if they're lost in the large websites and not all of them are interesting. By conducting this indexation audit, you can find issues such as your key pages/your business pages not being indexed at all, and other pages that are not useful for SEO being indexed.

It's integrated with content strategy because you can decide whether you should keep updates or prune your content pages. Technical SEO, of course, because you can work on improving your site architecture and internal linking to help Google rediscover the key pages.

This audit's like the first step. It's not always done properly because, as I said, you don't always have the tools. We didn't know about this ‘130 days’ rules before, so using server logs wasn’t always giving us this type of insight.

Basically, for large-scale websites like e-commerce, it was almost impossible before. Now, we have this insight, and it's transforming server logs analysis into a scalable and evidence-based method to detect the indexation. This is essential in today's SEO landscape.”

You mentioned that, sometimes, key pages wouldn't necessarily be indexed.

Why is this typically the case? Is it simply a case of not linking to them properly? Not having enough content on the pages? Or perhaps Google or other search engines are thinking that they're irrelevant and shouldn't be ranked?

“In the first place, maybe Google never discovered the page. Google is blocked because of robots.txt, meta robots, bad internal linking, or JavaScript – things like this.

If Google crawled the page, but it’s still not indexed, maybe the quality is poor, so it's not matching Google's quality criteria. There can be different reasons, starting from crawlability to content quality. Some pages can be indexed for a while, but then Google starts de-indexing.

Lately, we've seen (Adam Gent published studies about this) Google conducting a large de-indexation process to clean up its index, and pages not matching the quality criteria simply went out of the index.”

Can you use a tool like Oncrawl to determine the likely reason why a web page may not be indexed?

Or is this really a human being that has to analyse and step in after realising that it's not being indexed?

“You can use a tool, a log analyser like Oncrawl, to run this audit. Maybe I can start with this.

On one hand, you need to collect your server logs. On the other, you need to run a crawl on your website because you need to process the logs and detect Googlebot hits (so, visits from Googlebots), and filter on a period of 130 days, so you can list pages visited by Googlebot over this period. Then you cross-reference this data with a crawl of all the pages that can be discovered on your website, so you can identify the ones not visited over this period. This way, you find pages that are not indexed.

You can use any server log analyser tool or crawler to conduct this audit. Manually would be a bit hard, depending on the size of the website and especially the size of the server logs. But then, how to find why a page is not indexed, that's the next step.

To go further, there are different methods. First, there’s segmentation, so segmenting your pages by page type, to see if it's concerning a specific page type/a specific section on your website, or if it's randomly across the entire site. You can also segment using other criteria, such as publishing date, the size of the content, and quality criteria that you would set on your page, and try to find out why those pages are not indexed. Find the correlation.

It could be related to internal linking, content quality, duplication, etc. There are many examples. The thing is, you need to segment and try to find out why. Reasons are going to be different from one vertical to another, and from one type of website to another. E-commerce doesn't have the same issues as media publishers. I cannot give a generic answer for this.”

What about log files in general? Why is it so important to incorporate log files as part of the crawl instead of just asking your tool to actually crawl the website without them?

“Yes, so a crawler is going to pretend to crawl a website just like Google would – so, following the links, following the redirections, 301s – collecting data on each page to know whether the page has a meta robots or canonical tag, etc. It's going to give you a view of what Google is supposed to see, but you don't know for sure what Google sees.

This piece of information is actually in your server logs, because every time somebody (a bot or a visitor) asks to browse a page, it writes a line in the server logs. In this line, you know which pages were requested at what time, what the answer was, if the server actually responded, and you know the user agent. So, you can see whether it was a Googlebot or another bot (it could be OpenAI's bot, Bingbot, etc.).

If it's a visitor, because by checking the referrer, you can know what the previous page was. If it's someone coming straight from Google, you know it's a visit from Google. For someone coming from ChatGPT, it will be a click in ChatGPT leading to your website. All that information is really useful to see what really happens on the website. It's like analytics data, but for bots and not only visitors.”

What does this ‘130 days’ rule mean for when you would want to schedule these crawls? Would you want to do one of these crawls every 131 days?

“I was really surprised by this rule, but it also makes sense because Google is not going to expand the index. It's like there is one field to play, and the number of players is limited, but there are players waiting to get in. So, you need to get some of them out of the game.

What Alexis Rylko discovered is that, using the URL Inspection tool for a lot of websites he was auditing (he's working for an SEO agency), you can check the last crawl date for each page using that tool. If the website is not too big, you can actually use the URL Inspection tool; it will work fine. Next to the last date of a visit from Googlebot, you have the status of indexation – this information is given by Google.

By auditing multiple websites, he saw that after 130 days (if it's over 130 days or older than this), the state is always ‘not indexed’. It doesn't mean the page was never indexed; it means it's not in the index anymore. Adam Gent went further in the study, and he has access to even more data, and he found out that this rule was most likely to be true.

Your question was about whether you should do that every 130 days? No, actually not, because it's like floating. If I do the audit today, I need the last 130 days starting today. I wouldn't run such an audit every day or every week. Every semester or every quarter would be good enough because then you need to investigate why pages are de-indexed, if the ones that need to be indexed are currently indexed or not, and take actions to fix the issues.

It's like a technical SEO audit. It is something you run twice a year or four times a year, depending on your vertical.”

You also talk about the importance of bypassing the URL inspector tool API limits for large websites. What is this and why is it important?

“Because, with 2,000 queries per day, it's simply impossible for an e-commerce with millions of pages to know and audit the indexation state for every page – every product page, every listing page, etc. Using your logs, it's a lot of data, but tools can handle that.

This inspection API gave us the rule, so we could discover this rule, but using the logs is simply the way to do it at scale with real data.

It's your own data as well, which matters as well, and there is no sampling. It's really data that you own, and you can use it for analysis and do a lot of things with it.”

Jérôme, what is the key takeaway from the tip you shared today?

“The takeaway is that log analysis is not only about enterprises, your large-scale websites, and it's not only about crawl budgets.

It also offers you now, not an easy, but a scalable way to audit indexation – and this is super cool.”

Jérôme Salomon is Senior Technical SEO at Oncrawl. Find out more over at Oncrawl.com.