Organic search drama of the week, Google vs every rank tracker

On Monday 28 September a client’s weekly report told me their site had dropped out of Google on its own brand name. The report pulls live rankings through API connections, and that day the results page it got back was mostly forums. The client was not in the top 20. Search Console had the same search at an average position of 1.0 to 1.1 every day that week.

Search Console was right, this time.

Earlier that morning the same request from the same US location had returned a normal page, with Google reporting 186 results. The bad pull reported about 59,200.

Turns out the whole industry had the same Monday

The same day Charles Floate posted on X that Ahrefs, Semrush and “ALL the rank trackers” were way off from real Google. His screenshot of a tracker’s SERP for “charles floate” shows Reddit, LinkedIn, X and YouTube, and his own site is not in the top 13. Real Google puts charlesfloate.com first. The tracker’s position chart drops in every bucket on the same day: 25 fewer number one positions, 75 fewer in positions 11 to 30.

The replies under his post tell the same story. Some people saw it start around 19 September. Floate’s own read is that Google tested it for a couple of weeks and switched it fully on in the 48 hours before his post. An SE Ranking user said about 80% of rankings vanished from their system from 26 September. One SEO “almost had a heart attack thinking we lost tons of rankings LOL”. Someone running their own scraping scripts got 429 errors, which is Google saying too many requests. Another person got the junk page as a human, in incognito behind a VPN: only four sites on the page, and sometimes the real results after a few refreshes.

That last one points at where the query comes from, not at the tools. On 29 September the Japanese SEO blog seo_sub compared a private phone search with a desktop search through a proxy for a credit card query. The two top 10s had zero URLs in common.

Advanced Web Ranking published an explanation on 1 October. Their claim is that when Google suspects a query is automated, it can serve “a simplified page” where commercial and content-dense sites sit lower or drop out. Google has not confirmed that.

Previously on Google vs the scrapers

• January 2025: from 15 January Google stopped showing results without JavaScript. Before that a tracker could request the results page like any web page and read the rankings straight from the HTML. Now the results only appear after the browser runs Google’s script, and that script also checks whether you are a human or a bot. Google calls it SearchGuard. Every tracker had to start running a real browser for every query, which is a lot slower and more expensive. Google said it was about bots and spam and that fewer than 0.1% of searches are made with JavaScript off. Semrush and SE Ranking stopped refreshing, and Search Engine Journal called it a global tool outage.
• September 2025: the num=100 parameter went away. Trackers used it for years to get 100 results in one request. Now they get 10, and most of them broke.
• 19 December 2025: Google sued SerpApi, saying it made hundreds of millions of searches a day and got around SearchGuard.
• June to August 2026: result links turned into google.com/goto redirects. Derek Perkins from Nozzle says resolving every link on a five-page ranking pull takes 500 to 1,000 requests. Google confirmed it on 26 August.
• 20 July 2026: the court dismissed Google’s DMCA claims against SerpApi. Plain search results are not copyrighted works. Google got 21 days to refile a narrower claim, so it isn’t over.
• Mid September 2026: blocking got harder again. Nozzle got about 80% less data, and Sistrix and other data providers reported problems too.

So far the courts haven’t stopped scraping. Making scraped results worse is the cheaper option, and 28 September looked like that to me. That is my reading, not Google’s. Floate’s theory is that Google is protecting its own scraped data and its AI answers.

The second wrong number that week

The same data source said 33 of the client’s blog posts ranked for nothing. It made it into a draft report. When Search Console was connected, 20 of the 32 posts we could check had impressions. One had 16,600 impressions and 307 clicks.

A rank database only knows the keywords it tracks, and usually one URL per keyword. “No rankings found” does not mean “no traffic”.

Two wrong claims in one week, both caught before they reached the client. Fun week.

Search Console blends AI answers into search

Search Console beats a tracker because it is Google’s own data, not a scraped sample. But it now counts regular search and AI answers in the same numbers.

Google’s help page says AI Overviews and AI Mode impressions and clicks are part of the normal Web search type report. An AI Overview takes one position on the results page, and every link inside it gets that same position. If your page is the seventh source in an AI Overview at the top of the page, Search Console logs position 1. In AI Mode every follow-up question counts as a new query.

In June Google added a separate generative AI report to Search Console, and on 31 August it reached every site. It has impressions by page, country, device and date, with data from 18 May 2026. No queries, no clicks, no position. Google says the same data also stays in the overall report.

So you can see how many impressions came from AI, but you can’t take them out of any single query. A query’s average position is a mix of blue links and AI answers, and the report doesn’t tell you the ratio.

Optimizing organic visibility right now is a bit of a Magic 8 Ball. Reply hazy, try again. The tracker may show you a page no human sees, and Search Console blends two different things into one number.

Track the money

I have always said SEO should be run as arbitrage. Every page costs something to write and keep up to date, and every page either brings in money or doesn’t. You make more of what earns more than it costs, and you stop doing what doesn’t. Rankings and impressions are steps on the way. Google can’t serve revenue differently to a scraper, and revenue doesn’t get mixed with AI answers.

In practice: tie every sale and every lead to the landing page the visitor came in on, from analytics, the store’s orders or the CRM. Count euros per page per month and compare it with what the page cost. Make more of the pages where the difference is positive. For pages that earned nothing in six months, ask why they exist.

Revenue doesn’t show everything either. Someone who read about you in an AI answer and came back directly a week later isn’t tied to any page. It is still the only number nobody has blended.

What I do now

The weekly report checks itself. If the tracker puts the client’s own brand search outside the top 3 while Search Console has it in the top 3, that week’s rankings are left out completely, with one line saying the check looked faulty. Nothing about pages that “rank for nothing” goes into a report without Search Console data.

What to build more of and what to fix gets decided on revenue. Rankings and impressions tell me where to look.

Yesterday I wrote about Google’s September spam update, where tracker volatility on 25 to 27 September was one of the few signs of impact. I don’t know how much of that volatility was the update and how much was bad measurement. Nobody does yet.

If your tool shows a crash

Search your own brand name in an incognito window first. If you are on top there and not in the tool, the tool is wrong.

Open Search Console before you tell anyone about a drop. If clicks held, nothing happened to your site.

Then check revenue by landing page. If sales and leads come from the same pages as before, nothing changed.

Compare weeks, not days. One day of scraped data can be one bad sample.

Keyword difficulty is calculated from what the tool sees on the results page, so as Floate pointed out, that is off too. Search volumes come from other sources.

For competitors you have no Search Console, so read their tracked rankings as a direction for now, not an exact number.

Mari Luukkainen

Slop Cannon

Ex-VC, full-time builder. Personal record: 349 GitHub commits in one day. Find me on GitHub and at mariluukkainen.com.

Read more about Mari Luukkainen