Your Bot Traffic Logs Are Lying to You

TL;DR: Most of what shows up in your server logs as AI crawler traffic is fake. One operator verified that 81.8% of “AI assistant” visits were spoofed, and 87% of Googlebot requests came from imposters. If you’re making decisions about SEO or AI visibility based on raw log data, you’re working from fiction.

The Problem Nobody Is Checking

When a bot fetches your page, it announces a name. ChatGPT-User. ClaudeBot. Googlebot. Your server writes that name into the log, your analytics counts it, and you draw conclusions. The problem: that name is a self-reported string in the request header. Anyone can write anything there. Claiming to be Googlebot costs nothing and proves nothing.

Duane Forrester recently launched a new platform, CitationIQ.com, and spent two weeks auditing exactly who was actually crawling it. He expected modest numbers given zero promotional spend. What he did not expect was that most of even those modest numbers were fabricated. Of 33 requests claiming to be live AI assistant fetches, only 6 were real. Of 799 requests carrying the Googlebot name, only 107 came from a verified Google IP address. The rest were strangers wearing someone else’s uniform.

This matters for any operator spending real money on search visibility. Whether you’re running forex acquisition campaigns or pushing content for a regulated vertical, if your performance reporting is built on inflated crawler counts, you’re optimizing against fake signals.

How to Actually Verify Bot Traffic

The verification method is not complicated. Every major AI operator publishes the actual IP address ranges their bots use. A request is legitimate only if two things are true: the announced name matches, and the originating IP sits inside the published range. Name is the claim. IP is the proof.

The published verification lists are public right now:

• ChatGPT-User: openai.com/chatgpt-user.json
• Claude (all bots): claude.com/crawling/bots.json
• Perplexity-User: perplexity.com/perplexity-user.json
• Googlebot: developers.google.com/static/crawling/ipranges/common-crawlers.json
• CCBot: index.commoncrawl.org/ccbot.json

Forrester built his check in roughly 15 lines of Python using only the standard library. The logic maps each bot name to its published IP list, then returns one of three outcomes: verified (IP is in the published range), spoofed (ranges loaded and the IP is not there), or unverifiable (couldn’t determine, either because a list failed to load or the record was missing). Critically, he never calls something fake just because he couldn’t confirm it. That discipline matters, and the CCBot investigation below shows exactly why.

Operators running any kind of content-led acquisition, from iGaming player acquisition to mass tort lead generation, should be running this same check quarterly at minimum. Inflated bot counts skew crawl budget analysis, content performance benchmarks, and server load estimates all at once.

What the Spoofed Traffic Was Actually Doing

The fake AI assistant visits gave themselves away by where they went. A real assistant fetch lands on a real page. The spoofed ones, still wearing ChatGPT-User as their identity, went hunting for .env.production, secrets.yaml, and config.json. These were credential scanners borrowing a trusted name to slip past server-side filters, and the IP verification caught every one of them.

The Googlebot impersonation problem is even older and more entrenched. Googlebot has been the most-spoofed crawler name on the web for close to two decades, which is exactly why Google tells webmasters to verify by IP rather than trust the header string. On a brand-new site with essentially no traffic, 87% of Googlebot-labeled requests were fakes. Some were running Googlebot user-agent strings tied to products Google retired years ago, copied from old lists and never updated.

The practical takeaway: your Googlebot line in Google Search Console is not what you’re seeing in raw server logs. The logs show “claims to be Google.” Search Console shows verified Google. The gap between those two numbers can be enormous, and if you’re doing any manual log file analysis to supplement your analytics stack, you need to account for it.

Retrieval vs. Training: Two Different Games

Once you strip the fakes out, what the verified crawl data reveals is more strategically interesting than the spoof rate. AI crawlers are not all doing the same thing, and lumping them together is a mistake.

Some crawlers do retrieval. They build the index that gets pulled into an answer today during a live user session. If someone asks an AI assistant a question right now and the assistant surfaces current sources, retrieval crawlers are the machinery behind that. Retrieval is about whether you show up this week.

Others do training. They harvest content that gets folded into the weights of the next model. When a training crawler takes your page, you won’t see that in referral traffic. It’s a deposit into a corpus that will shape how models answer questions for years, often without ever fetching you again. The payoff is delayed, compounding, and invisible to every standard dashboard.

On Forrester’s domain over 14 days, the most active verified crawler was not Google. It was Anthropic’s ClaudeBot at 166 confirmed crawls, ahead of verified Googlebot at 107, with OpenAI’s GPTBot at 46. Two weeks on one small site is not a trend, but the composition matters. Who is spending crawl budget on a new, unpromoted domain is the kind of signal that becomes strategic once the volume is real.

Operators building long-term content plays, particularly in high-CAC verticals where law firm lead generation or crypto acquisition depends on brand authority, need to think about both signals separately. Being retrieved gets you traffic this month. Being trained on determines whether AI systems know you exist in two years.

The CCBot Investigation: How to Chase Unverifiable Rows

Common Crawl’s CCBot is arguably the most consequential training crawler on the web. It produces the open dataset that sits underneath a large share of recently trained models. So when the automated check returned 0 verified, 4 spoofed, and 16 unverifiable CCBot requests, the 16 required manual investigation.

Unverifiable does not mean fake. It means go find out. Forrester ran four independent checks:

1. Published IP list: None of the 20 CCBot-labeled requests fell inside Common Crawl’s published ranges.
2. Reverse DNS: Real CCBot resolves to a commoncrawl.org hostname. Four resolved to something else entirely. Sixteen had no reverse record at all.
3. Corpus index: Common Crawl runs a public index where you can query whether a domain has been captured. Three recent monthly crawls, wildcard search across all paths. Nothing.
4. WHOIS: Every raw IP traced to commodity hosting infrastructure spread across multiple countries, the cheap rented servers that scanner operations run on.

Four independent angles, one answer: all 20 were impostors. The lesson is that your automated check should refuse to call something fake just because a DNS record is absent. An absent record is not evidence of fraud. But it is an invitation to dig, and the digging resolves the picture. Any operator doing a proper marketing audit that includes technical SEO and crawl analysis needs this step in the process.

The Gemini Problem: A Familiar Dead End

There is one major AI player you cannot measure at all by design: Google Gemini. OpenAI, Anthropic, and Perplexity each expose distinct, verifiable signals. You can separate their training crawler from their retrieval crawler from their live assistant fetch, and confirm each by IP. Google does not work this way.

There is one Googlebot crawl. Whether content it gathers feeds Gemini training is governed by a robots.txt token called Google-Extended, which is not a crawler. It is a permission flag on a crawl that already happened. There is no Gemini fetcher in your logs because Google did not build one. You can confirm Googlebot. You cannot confirm anything past it.

This is structurally identical to what happened in 2011 when Google encrypted search referrers and keyword data collapsed into “(not provided).” The granularity went away and operators were handed a flag in place of a measurement. The AI era is doing the same thing. Where competitors expose training, retrieval, and demand as separate verifiable events, Google bundles them into a single crawl and an invisible permission token.

For operators running paid performance campaigns alongside organic, this blind spot matters. You can optimize for Gemini visibility in theory but you cannot measure whether it’s working in any direct way. Investing in precision audience targeting on paid channels becomes a more accountable alternative while organic AI visibility for Google remains a black box.

What This Means for Performance Marketing Operators

The operators most exposed to this problem are the ones doing the most sophisticated measurement: agencies and in-house teams running content programs at scale in high-CAC verticals, tracking every traffic signal as a potential input into acquisition modeling.

Here is what to do this week. Pull a date range from your server access logs. Match the bot name strings against the published IP ranges from OpenAI, Anthropic, Perplexity, Google, and Common Crawl. Record verified, spoofed, and unverifiable separately. Then look at your Googlebot line and prepare for the number.

When you hit unverifiable rows, do what Forrester did with CCBot: pull the raw IPs, run WHOIS, check reverse DNS, and query the corpus index if it’s a training crawler. The picture will resolve. The method takes an afternoon. The alternative is building acquisition strategy on fabricated crawl data, which is no different from optimizing paid spend against fraudulent click traffic.

If you’re running a content program that feeds crypto lead generation or any other regulated acquisition funnel, the signal quality of your organic channel data is not a technical footnote. It’s a business risk. Verify the baseline before you trust the trend.

Originally reported by Search Engine Journal, June 2026.