You Cannot Rank in a Slot Machine

Ask ChatGPT the same question twice. Not a trick question. Something boring, like “what are the best fleet maintenance software options for mid-size trucking companies.”

You will get two different answers. Different vendors, different order, sometimes a company in answer one that is completely absent from answer two. Ask it from a different account, or an hour later, or with one extra word in the prompt, and the list shuffles again.

Now go look at the AI visibility report someone just sold your team. It says you are ranked #4 for that query. Four. A whole integer. Tracked daily, plotted on a line chart, trending up 12 percent this month.

That number is not measuring your visibility. It is measuring one sample from a probability distribution and then dressing it up as a position on a leaderboard.

Rank was always a fiction. It is now a worse one.

To be fair, classic search rank was never as solid as the reports implied either. Personalization, location, device, logged-in state, testing buckets. Two people in the same office got different results for the same query in 2019. Everybody in SEO knew this and everybody quietly agreed to keep pretending, because a rank number was legible to executives and “a distribution of possible positions weighted by user context” was not.

The pretending was survivable back then, because the variance was small and the underlying index was stable. Google’s results for a given query on a given day were mostly the same page for most people. You could average across a bunch of samples and get something that meant roughly what you thought it meant.

Answer engines broke that. The output is generated, not retrieved. There is a temperature setting. There is a retrieval step that pulls a different slice of sources depending on how the model decided to expand your question this time. There is memory of the conversation you had four turns ago. There is a model version that gets swapped under you on a Tuesday with no announcement, no changelog anyone outside the company reads, and a completely reshuffled sense of who the credible vendors are.

You are not being ranked. You are being sampled.

What the tools are actually doing

Here is the honest description of most AI visibility products right now. They maintain a list of prompts you care about. They run those prompts against a few models on a schedule, from their own infrastructure, with their own accounts, in their own geography. They parse the text output for your brand name. They count.

That is a real thing to do. It is not nothing. Directionally, if you go from being mentioned in zero out of fifty runs to twenty out of fifty, something changed and it is probably good.

The problem is what happens between the counting and the slide deck. The count becomes a rank. The rank becomes a trend line. The trend line becomes a quarterly goal. And now three people have a bonus attached to a number that moves when a model provider ships a checkpoint, which is roughly like tying compensation to the weather.

Ask the vendor two questions and watch the room get quiet:

• How many times do you run each prompt before you report the number?
• What is the variance across those runs?

If the answer to the first is “once,” you are buying noise with a logo on it. If they cannot answer the second at all, they have never looked, which tells you they are selling the chart and not the measurement.

The thing that is actually measurable

Here is the part that annoys people, because it is less fun than a leaderboard.

You cannot reliably observe your position inside someone else’s model. You can observe what happens on your side of the wire. Sessions arriving with an AI assistant as the referrer. Which pages they land on, which is usually not your homepage and is often some deep documentation page you forgot existed. Whether those sessions convert, and at what rate compared to everything else. Whether the pipeline they create closes, and at what size, and how fast.

That data is yours. It sits in your analytics, your CRM, and your billing system right now. It does not shuffle when a model gets retrained. It is boring, it is countable, and it answers the only question an executive actually has, which is whether any of this produced money.

The catch, and it is the whole catch, is that answering it requires those three systems to agree on what a customer is. In most companies they do not. Web analytics has sessions. The CRM has contacts and accounts, entered by a human at 4:50 on a Friday. Billing has invoices under a legal entity name nobody else uses. Nobody can join them without a person exporting three CSVs and doing surgery in a spreadsheet, so nobody does it, so instead everybody stares at the rank chart, because the rank chart is there and it has a number on it.

That is the real reason these tools sell. Not because the measurement is good. Because it is available, and the good measurement is locked behind a data integration nobody wants to own.

What to do on Monday

Keep the visibility tool if you already bought it. Just demote it. It is a smoke detector, not a scoreboard. It is useful for “are we present in this conversation at all,” it is useless for “are we number three,” and anybody who reports it to a board as a position is going to get asked a question they cannot answer.

Stop setting numeric targets on it. A goal of “reach an average visibility score of 62” is a goal to chase variance. Set the target on the thing you control instead, which is whether the source material exists: does a page on your site actually answer the question a buyer asks in that prompt, in plain language, with specifics a model can quote. That is not a growth hack. It is just writing the thing down properly, which is unglamorous enough that most of your competitors will skip it.

And then go do the unsexy work of connecting your analytics to your CRM to your revenue, so that when AI referral traffic shows up you can say what it was worth instead of what position you allegedly held.

The companies that win this are not the ones with the best AI search dashboard. They are the ones who can tell, at the end of the quarter, whether any of it mattered.

If getting those three systems into one place is the part that keeps not happening, that is the problem THE DASHBOARD exists to remove. No rank charts. Just your own numbers, in one place, where you can ask them things.