AI Search Reads Your Site — But Can It Understand It?

TL;DR: An audit of 50 major websites found the average AI readiness score sits at 56.6% — and that average masks a near-total collapse at the attribution and agentic layers. Most operators have cleaned up their crawlability. Almost none have given AI the context to understand what their content actually means, and fewer than 5% are ready for AI agents to transact on their sites. Here is what the data shows and what to do about it.

Crawlability Is Not Understanding

The framing most marketing teams operate with goes like this: if AI bots can reach the content, the content will be cited. That is a reasonable first principle, but it is incomplete. The analogy that holds up is a child learning phonics. A six-year-old can read words aloud with reasonable fluency while absorbing almost nothing of the meaning. Reading is one layer. Comprehension is a separate layer on top of it. And what you do with that comprehension — the ability to act on it — is a third layer entirely.

Search Engine Journal’s audit of 50 major websites across retail, SaaS, travel, publishing, and finance used an instrumented browser to capture live HTTP responses, rendered DOM, raw server HTML, and machine-discovery endpoints — all on the same day. The framework scores 12 established technical signals across three layers: retrievability, attribution and meaning, and agent transaction and discovery. The highest score in the cohort went to Airbnb at 79.2%. The mean was 56.6%. That is not a passing grade when your competitors are in the same race.

Layer One: Retrievability (Average 74.4%)

This is where most SEO teams already operate. The eight established signals here — robots.txt AI user-agent directives, ARIA labeling, semantic HTML, token-efficient DOM density, server-rendered HTML, sitemap declaration, accessibility tree integrity, and form machine usability — overlap heavily with conventional technical SEO. Most sites perform reasonably well. Only three of the 50 audited sites scored below 50% on retrievability.

For high-budget operators running paid acquisition at scale, this layer is table stakes. If your technical SEO house is in order, you are probably already hitting the basics. The problem is that operators are treating retrievability as the finish line when it is closer to the starting blocks.

Robots.txt AI user-agent directives are part of this layer — but there is a meaningful difference between a standard robots.txt that passively permits crawling and the more strategic Content Signals Policy that Cloudflare introduced, which lives in a separate section of this article.

Layer Two: Attribution and Meaning (Average 38.5%)

This is where scores fall off a cliff. The average drops from 74.4% at the retrievability layer to 38.5% at attribution. Two established signals sit here: JSON-LD schema and Content Signals Policy.

JSON-LD structured data was present on the homepage of 35 out of 50 sites — 70%. That sounds reasonable until you consider that the remaining 30% have given AI no structured signal about what their content means. Schema is how AI distinguishes between the product price and the loyalty discount price. It is how AI determines whether an article is about Jaguar the car brand or Jaguar FC. Without it, AI makes its best guess — and that guess can be wrong, producing hallucinations or brand misrepresentation in AI-generated answers.

The more alarming signal: only 5 of the 50 audited sites have implemented Cloudflare’s Content Signals Policy. This is a set of directives that specify exactly what AI crawlers are permitted to do — index for search, respond to live queries, train on content. Without it, your AI strategy is a binary: block everything or allow everything. Operators running regulated verticals like iGaming acquisition or forex lead generation cannot afford that binary. Content that is fine for a search index is not always appropriate for model training, and the Content Signals Policy is currently the only established mechanism to make that distinction at the server level.

A structured technical marketing audit at this layer alone — schema coverage and Content Signals Policy implementation — would surface gaps most operators do not know they have.

Layer Three: Agent Transaction and Discovery (Average 2.1%)

The number is not a typo. Of 48 sites where endpoint testing was possible, 46 scored zero on agent transaction readiness. Two sites — Airbnb and Vercel — have implemented OAuth authorization server metadata. Neither has implemented OAuth protected resource metadata. Both eBay and Wikipedia block same-origin fetch via CSP.

Agentic browsers and agentic commerce have only been in production since late 2025. The audit identifies 13 signals in this layer, of which only two are classed as established. OAuth authorization server metadata and OAuth protected resource metadata are the two. Together, they make it possible for an AI agent to authenticate with your site and transact on a user’s behalf — book a reservation, submit a lead form, initiate a trade, process an application.

Operators investing in AI-powered lead qualification are already thinking about this layer even if they have not framed it in these terms. An AI agent that can reach your site, understand what you offer, and complete a conversion action without human intervention is not a distant scenario. It is where the infrastructure is pointing. The 2.1% average means almost no one is ready — and first movers will have a structural advantage that compounds over time.

What This Means for Performance Marketing Operators

Operators spending $10K or more per month on paid acquisition are already paying for precision. Precise audience targeting drives cost-per-lead down on the paid side. But if AI search is increasingly the channel through which high-intent users are making decisions — and the data on AI Overview adoption suggests it is — then the organic side of that equation has new technical requirements that most teams are not meeting.

Three concrete actions translate directly from this audit:

Schema coverage across all commercial pages, not just the homepage. The audit only tested homepages. Most schema gaps are deeper in the site — product pages, service pages, location pages. JSON-LD should be present and semantically rich on every page that carries commercial intent. For operators in legal, this means structured data that distinguishes practice areas, attorneys, and case types. For crypto exchange operators, it means schema that correctly identifies token types, pricing, and regulatory disclosures.

Content Signals Policy implementation. This is a 2025-era signal that 90% of major sites have not implemented. For operators in any regulated vertical, this is not optional infrastructure — it is risk management. Set your AI crawl permissions explicitly rather than leaving them to the default interpretation of each platform’s bot.

OAuth infrastructure scoping. Layer three readiness does not require full agentic commerce deployment today. It requires scoping what OAuth implementation would look like for your site so that when agentic browsers become a meaningful acquisition channel — and the trajectory suggests 2027 is a reasonable estimate — you are not starting from zero.

Operators in CDL recruitment, for example, are already dealing with high-friction application flows. CDL driver recruitment funnels that support agent-assisted form completion will outperform those that require fully manual interaction, particularly as AI assistants become the default research tool for job seekers. The same logic applies to personal injury law firm intake — AI agents that can identify the right practice area, qualify the case, and pre-fill intake forms will shift lead volume toward firms with Layer 3 readiness.

The Audit Framework as a Diagnostic Tool

The 27-signal framework described here — 11 signals in retrievability, 3 in attribution, 13 in agent transaction — is not a replacement for vertical-specific technical SEO. The researchers note that in client engagements, they weight the scoring by industry. A SaaS company with no transactional commerce has different Layer 3 priorities than an iGaming operator or a legal intake platform.

What the framework does provide is a clear structure for identifying where effort is being concentrated and where it is absent. Most operators are spending the bulk of their technical SEO time in Layer 1, where the gains are smallest and the competition is most dense. Layer 2 and Layer 3 are where competitive separation is available right now, at a fraction of the effort required to claw out ranking improvements in traditional search.

The mean score of 56.6% across 50 major websites is not a benchmark to beat — it is a floor. Operators serious about AI search visibility need to be running structured diagnostics across all three layers, not just auditing crawl accessibility and calling it done.

Originally reported by Search Engine Journal, September 2026.