Invisible Authority: I built a tool that measures information gain automatically

How do you know if your content adds information gain?

You check your draft against the pages already ranking. Break each ranking page into individual claims, split them into the consensus everyone repeats and the claims only one page holds, and see what you add on top of that shared core. That surplus is your information gain, and if you're only restating the consensus, you have none.

Doing that by hand takes hours per keyword, so I built a tool with Claude Code that does it against the live SERP. Most of my SEO work runs through Claude Code these days, so measuring this instead of writing another explainer was the natural move.

I ran it across 15 queries, 97 pages, over a thousand claims. Only about 14% of the claims were consensus, and the rest sat scattered, each site holding a different piece. One honest surprise came out of it: being more unique didn't push pages higher, it just got them in the door. Below is the method, the numbers, and the prompt and script to run it yourself, including a look at the "information gain seo" results, which prove the point better than I planned.

What is information gain?

Information gain is a concept from a Google patent. It scores a page by how much new information it adds on top of what the reader already saw in the results ranked above it. If the first five pages all say the same thing and the sixth repeats it, the sixth has near-zero information gain. The page that introduces a fact, a number, or an angle nobody else covered scores high.

Information gain: a Google patent concept for scoring documents by the novel information they contribute relative to results the user has already seen. The term also comes from information theory and machine learning, where it measures how much a split reduces uncertainty. Same root idea, different field.

Semrush has a solid breakdown of it, and their definition is a good one:

Information gain in SEO is a metric that Google may use to evaluate the uniqueness of your content compared to similar content the user has already viewed. If your content has original and useful elements, the theory is that it may rank higher in secondary search results (results retrieved after initial results to enhance relevance and diversity).

One caveat before anyone quotes me wrong. Information gain is a patented concept and a widely discussed signal, not a confirmed live ranking factor I can hand you proof of. Everything below is my own experiment. The idea makes a lot of sense to me, and the data is interesting, so I'm sharing the method and the numbers.

The gap I found while researching this exact topic

Before writing this, I ran my own tool on the keyword "information gain seo." Nine articles hold page 1. Semrush, Animalz, Backlinko, Search Engine Land, Clearscope, Searchbloom, and a few more. Good writers, real domains.

Every one of them defines the concept and gives the same three pieces of advice: add original data, add expert quotes, build a unique framework. The AI Overview at the top compresses all nine into exactly those bullets. Novelty, specificity, extractability.

So the entire first page tells you information gain matters and that you should add unique content. Almost none of it shows you how to find what's already been said, or how to measure whether your draft actually clears the bar. That's the gap. A page full of theory, and a missing tool. Which is the whole reason this article exists, and the reason I built the thing I'm about to describe.

Why does information gain matter for AI search?

Two reasons, and the second is the one I care about most.

Google rewards a page that moves the topic forward instead of echoing what already ranks. That's the premise of the patent, and it lines up with how the helpful content work has trended.

AI citations run on the same logic, harder. When ChatGPT, Perplexity, or an AI Overview assembles an answer, it pulls from the source that holds the piece the others are missing. Cover only the consensus and you are one interchangeable page among ten, quotable by nobody in particular. Hold a claim nobody else has and you become the source the model has to reach for. In an AI answer there is one citation for a given fact, not ten blue links. The page with the unique claim wins the slot.

So for any page you build, two questions decide its fate. What does everyone already say, which you have to cover to be credible. And what can you add that nobody else has, which is your information gain.

The problem: measuring it by hand is brutal

Doing this properly means reading the eight to ten pages that rank, listing every factual claim on each one, then cross-checking which claims are shared and which appear only once. For a single keyword that's a couple of hours of tedious pattern-matching. Multiply by a content calendar and it never gets done. So it stays a nice principle nobody actually operationalizes.

That's exactly the kind of work I'd rather hand to a machine, so I built one with Claude Code.

How does the tool work?

Four steps, fully automated:

Pull the live top organic results for the keyword.
Read each ranking page and break it into atomic claims.
Cluster claims that say the same thing across different pages.
Split the clusters into consensus and unique, and write it up as a brief.

DataForSEO: the API behind step 1. It returns the live Google results page as clean structured data: positions, domains, URLs, the featured elements. The tool works off the real SERP instead of asking a chatbot to guess who ranks, which it will happily do wrong. Pay-as-you-go, cents per query.

Atomic claim: one self-contained, checkable statement pulled from a page. "A canonical tag goes in the head." "LCP should be under 2.5 seconds." Reducing pages to atomic claims is what lets you compare two sites that phrase the same idea in different words. Without it you're comparing prose, and everything looks unique.

The output is a content brief instead of a blank page. It shows the coverage floor every credible page has to meet, and the open space around it where your gain lives. Feed that into the writing and you're building from a map, not a hunch.

What I found running it on 15 queries

I ran the tool across 15 SEO queries, 97 readable pages, over a thousand atomic claims.

Only about 14% of the claims were consensus, meaning most of page 1 agreed on them. More than half of every claim appeared on a single page and nowhere else. The shared core is small, and around it sits a large pile of material where each site holds a different piece.

Then I ran it on a real client service keyword. Consensus dropped to zero. Not a single claim was shared by even 70% of the ranking pages. Every competitor covered something different, which means for that term the open space is the whole field.

The entire run cost me 50 cents of DataForSEO. The analysis itself ran inside my Claude subscription, so there was no model bill on top.

The honest test most "studies" skip

I could stop there and tell you uniqueness wins. Instead I checked whether the data actually says that.

If information gain were the ranking lever on its own, position 1 should be the most original page on the SERP. So I measured the correlation between rank and unique-claim share across all 97 pages.

It came out flat, even slightly the other way. Position 1 averaged about 32% unique claims. Position 7 averaged 40%. There's no clean line saying "more unique, higher rank."

What the data does support is a floor. Every ranking page, top or bottom, carried roughly a quarter to a third original content. Nobody made page 1 on pure rehash. So treat 25% unique as the price of admission, not a dial you crank for a higher spot. Automated content that chews up what already ranks sits near zero, which is exactly why it doesn't get in. Proving what earns the actual citation needs a different dataset, tracking which pages the AI engines quote, and that's the layer I'm building toward next.

How do you use the consensus and unique buckets?

The consensus is your entry ticket. The handful of things most pages say are the basics you cover to be a legitimate answer. If your draft already has them, that's a box checked, and a good sign.

The unique claims are your gain. Every claim only one competitor made is a gap the rest left open. First-hand numbers, a specific threshold, a named case, an angle nobody took. That's what an AI engine reaches for when it needs to say something the common sources didn't cover.

The workflow falls out of that. Cover the consensus so you belong. Consolidate the scattered unique claims into one place. Add your own first-hand material on top, the data or the experience only you have. The result carries more than any single competitor, because each of them started with one slice and you're holding the whole set plus your own.

Run it yourself

There are two ways to run this, and they use the same method underneath. Option 1 is a prompt you paste into any AI assistant, for a fast read on one keyword. Option 2 is a Python script that pulls the live SERP itself, for exact, repeatable numbers across a batch. Pick based on how far you want to go.

Option 1: the prompt (fast, no setup). Paste it into an assistant that can browse the web, give it your keyword, and you get the consensus-and-gaps split in a couple of minutes. The one limit: a model with no live search will guess who ranks and hand you confident nonsense, so run it somewhere connected to the web, or paste the competing pages in yourself.

You are analyzing Google page-1 results for one keyword to find information-gain opportunities.

KEYWORD: [your keyword]
COMPETITORS: [browse the live top 8-10 organic results, OR paste the full text of each ranking page below, labeled by domain]

Do this:

1. For each ranking page, extract up to 25 atomic claims. An atomic claim is a
   single, self-contained, checkable statement about the topic. Rewrite each one
   in neutral, brand-free language, under 25 words. Skip marketing copy,
   navigation, author bios, and calls to action. Split compound sentences.

2. Cluster claims that say the same thing across pages. Two claims belong
   together only if a reader who saw one would learn nothing new from the other.
   Different numbers for the same metric are different claims.

3. Output three lists:
   - CONSENSUS: claims made by most of the pages, with how many said each. This
     is the coverage floor a credible page must meet.
   - UNIQUE: claims made by exactly one page, labeled with the source domain.
     This is the scattered material to consolidate.
   - MISSING: sub-topics, questions, or data types none of the pages covered.
     This is open white space to own with original content.

4. Finish with a one-paragraph content brief: what to cover for the entry ticket,
   which unique claims to pull together, and what first-hand material to add that
   appears nowhere on page 1.

Option 2: the Python script (accurate, repeatable). It pulls the live SERP itself through DataForSEO, so the data is real and the run is repeatable: same input, same output, every time. The claim extraction runs inside your Claude or ChatGPT subscription, so there's no model bill. The only cost is the SERP data, 50 cents for my whole 15-keyword run. It takes a few minutes of setup and your own DataForSEO keys.

Download the script (information_gain.py)

Invisible Authority

I'm calling this Invisible Authority. The idea took shape out of a discussion in my SEO community this week, where the sharpest conversations I'm in about AI search happen to be. It tells you what to put on the page to earn the citation: the consensus you have to match, and the gain that makes you the one worth quoting.

It comes from a simple instinct. Stop guessing what search and AI reward, and measure it. This one cost me 50 cents to prove out. I'll be running it on client keywords from here on, and sharing what holds up.

If you want this run on your own keywords, or a site built to hold the citation, let's talk.

FAQ

What is information gain in SEO?

Information gain is a score for how much new, useful information a page adds on top of what the reader already saw in the results ranked above it. It comes from a Google patent (and, separately, from information theory). A page that only repeats the consensus scores low. A page that introduces a fact, a number, or an angle nobody else covered scores high.

Is information gain a confirmed Google ranking factor?

No. It's a patented concept and a widely discussed signal, not a live ranking factor anyone can prove. When I measured rank against unique-claim share across 97 ranking pages, the correlation was flat. What the data does show is a floor: every page on page 1 carried roughly a quarter to a third original content. Treat originality as the price of admission, not a dial you crank for a higher spot.

Should information gain be high or low?

For content you want found, high. The whole point is to add information the ranking pages are missing, so a search engine or an AI answer has a reason to reach for you instead of the ten pages that all say the same thing. (The "high is better" phrasing also comes from machine learning, where a high-information-gain split does a better job of separating the data. Same instinct, different field.)

How do you measure information gain?

Read the pages that rank, break each one into atomic claims, then sort the claims into consensus (what most pages agree on) and unique (what only one page says). Doing it by hand takes hours per keyword. I automated it: a prompt for a fast read on one keyword, and a Python script that pulls the live SERP through DataForSEO for exact, repeatable numbers across a batch. Both are in this post.