Benchmarks

Benchmarks measures AI brand visibility: when a shopper asks an AI model a buying question in the brand’s category, how often does the model name the brand — and which competitors come up instead? It’s the outside view of everything else Lumio does. Scoring and enrichment improve the data; Benchmarks tracks whether that work is moving the needle where it counts.

Benchmarks is part of the Elite plan. Open Benchmarks under Insights in the sidebar.

Visibility is scored per model. Perplexity answers with live web retrieval (“Grounded”); the others answer from trained knowledge (“Trained”).

How it works

1. Prompts are generated from the catalog

Lumio samples the catalog and generates a set of shopping prompts — the kind of question a real shopper would ask an AI assistant (“best waterproof hiking boots for wide feet,” not “tell me about Canopy”). The prompts deliberately never mention the brand, because the test is organic discovery: would the model surface the brand on its own?

The generated set is locked as a baseline. Every run uses the same baseline so the trend line stays comparable over time. Regenerating the prompts starts a new baseline.

2. Prompts run against four AI models

Each prompt is sent to ChatGPT, Perplexity, Claude, and Gemini. One important distinction: only Perplexity answers with live web retrieval. ChatGPT, Claude, and Gemini answer from their trained knowledge, so their results reflect what the model already “knows” about the brand and its category, not a live web ranking. Both signals matter — they’re just different questions.

3. Responses are judged for brand and competitor mentions

Each model’s answer is evaluated by Claude to determine whether the brand was mentioned, how (recommended, listed, compared, or spoken of negatively), and which competing brands showed up. The judging runs through Anthropic’s Batch API to keep the cost of a full run down.

Reading the results

• Visibility score — The share of prompts where the brand was mentioned, computed per model and then averaged across models. Responses that fail (a model or judge error) are left out of the math rather than counted as a miss.
• Trend — Because every run uses the same baseline prompts, the score is comparable run over run.
• Competitor visibility — The competitors on this page aren’t picked by Lumio. They emerge from whatever brands the models name in their answers. The table ranks the top competitors by share of voice, showing total mentions and reach (the share of prompts each appeared in).

Corrections

Model output is messy — a competitor might be named three different ways, or a model might refer to the brand by a parent company. Lumio lets the brand correct attribution with aliases: mark a name as the brand, mark it as not the brand, rename it, or hide it. Corrections apply as an overlay on past runs and feed into how future runs are judged, without altering the raw model data.

Running benchmarks

Runs can be scheduled weekly or monthly, or triggered manually. Manual runs are rate-limited per workspace, and only one run is active at a time. An email arrives when a run finishes, with the resulting visibility score.