AI engines barely agree, so one blended visibility score hides the truth

A new benchmark found that ChatGPT, Claude, Grok and Perplexity agree on who to recommend just 17% of the time when asked the same question, compared with 43% agreement when the same engine is asked twice. If you're tracking one blended "AI visibility score" across platforms, this data suggests that number is hiding the most important part of the story: which engine actually matters for your category.

What changed?

PR agency Crackle PR published its Q4 2026 AI Visibility Index on 20 October 2026, built from 500 answers collected by asking four AI engines the same 25 questions about choosing a B2B tech PR agency. Each engine's API was asked the same 25 questions, with web search switched on, five separate times.

The headline finding: average top-five shortlist overlap was about 43% when the same engine answered again, versus 17% across different engines. An engine is roughly two and a half times more consistent with itself than it is with a rival engine answering the identical prompt.

The study put a specific brand on this gap to make it concrete. Edelman, one of the world's largest PR firms, scored 22.0 out of 100 on Claude and 20.3 on Grok, but just 3.3 on Perplexity and 0.7 on ChatGPT. Averaged together, it lands in the second tier, and dropping either Claude or Grok from that average pushes Edelman out of the top five. A brand's position can swing from "winning" to "invisible" purely depending on which engines get folded into the average.

The researchers checked whether this was just noise. The same-engine versus cross-engine overlap gap sat at 25.9 points, with a 95% interval of 22.7 to 29.5 percentage points, from 1,000 question-level cluster bootstrap resamples. The direction held for top-three lists, full lists, a size-insensitive overlap measure, and with Crackle PR removed from the data. That last check matters, because Crackle PR produced the research and is one of the agencies in the benchmark, though it appears in the archive but is excluded from the ranking. The central result holds with Crackle removed, and public methods and archived answers allow scrutiny, though they don't remove the conflict of interest entirely.

This isn't an isolated finding. Separate research into citation behaviour has found similarly low cross-platform overlap: one analysis of 680 million AI citations found just 11% domain overlap between ChatGPT and Perplexity, while a separate study found 12% source overlap across 15,000 queries on three platforms. Different methodology, different industries, same conclusion: the engines are not drawing from the same evidence pool, so they rarely reach the same answer.

Why does this matter for your business?

If you've bought, or are considering buying, an AI visibility dashboard that gives you one overall score across ChatGPT, Claude, Gemini, Perplexity and Grok, this study is a warning that the blended number can mask which single engine is actually driving the result, good or bad.

It also changes how you should read a bad score. If your brand scores low on average but strongly on one or two engines, averaging in the engines where you're invisible can make a genuinely strong position look mediocre, exactly what happened to Edelman in the Crackle PR data.

The practical risk cuts the other way too. A brand could look strong on a blended score purely because it performs brilliantly on one engine that happens to carry a heavy weighting, while being essentially absent everywhere else that customers might actually be asking.

For marketing leaders reporting AI visibility up the chain, this means a single number in a board deck is doing more hiding than revealing. The question worth asking isn't "what's our AI visibility score" but "which specific engines are our buyers actually using, and how do we show up on each one individually."

What should you do now?

Start by breaking any existing AI visibility tracking apart by engine rather than looking at a blended average. A score of 40 out of 100 that's actually 80 on one platform and near-zero on three others needs a completely different response than a genuine, even 40 across the board.

Next, work out which engines your actual buyers use for research. A B2B software buyer researching vendors behaves differently from a consumer researching a holiday, and the engine mix shifts accordingly, so weighting your tracking towards the engines that match your audience matters more than chasing every platform equally.

Treat consistency, not just presence, as a signal. The 43% same-engine repeat rate in the Crackle PR study shows that even a single engine doesn't give identical answers every time it's asked, so one good result on one engine on one day tells you less than a pattern tracked over several runs and several weeks.

If you haven't checked where you currently stand across the different engines, running a free check of your own brand is a reasonable starting point, and Sited's free audit at sited.online will show you how you're showing up across the major AI platforms individually rather than as one averaged figure.

Finally, be sceptical of any vendor or report, including this one, that measures itself. The Crackle PR study is transparent about running its own agency through the benchmark, and while it shows the result holds with its own listing removed, that disclosure is exactly the kind of detail worth checking in any AI visibility report you're handed.

Frequently asked questions

What is the Crackle PR AI Visibility Index?

It's a quarterly benchmark that asks ChatGPT, Claude, Grok and Perplexity the same set of questions about choosing a B2B tech PR agency, then measures how much the four engines agree on their shortlists. The Q4 2026 edition was released on 20 October 2026.

Why do the AI engines disagree so much?

Each engine draws on a different mix of training data and live web search results, so they surface different sources and, in turn, different recommendations. Related research into citation overlap found comparably low agreement between platforms, including just 11% domain overlap between ChatGPT and Perplexity in one large-scale citation study.

Does this mean AI visibility tools are useless?

Not useless, but a single blended score across all engines can hide more than it reveals. The fix is to look at performance engine by engine rather than relying only on an averaged figure, particularly if your buyers are concentrated on one or two specific platforms.

Should I trust a study published by a company that's also in the ranking?

Treat it with appropriate caution but don't dismiss it outright. Crackle PR disclosed its own participation and showed the core finding holds even when its own listing is excluded from the ranking, which is a reasonable transparency standard, though it's still worth checking the methodology yourself.

How often should I check my AI visibility?

Given that even the same engine only repeats its own answer about 43% of the time, a one-off check isn't very informative. Running checks regularly, and across each engine separately, gives a far more reliable picture than a single snapshot.