How we measure what AI tells your buyers.
Every Arbling audit is built from recorded answers: the buyer questions we asked, the assistant that answered, the date, and what it said. This page explains how we ask, how we count, and how you can check any report yourself.
What we measure
We ask AI assistants the questions a buyer asks before contacting a supplier, then record which companies each answer names, which sources it cites and what it says about the company we audit. Nothing in a report comes from a model’s opinion of the company: every number is a count over recorded answers.
The questions
- Three buyer questions per company, phrased around what the company actually makes and where it sells, for example “Recommend the best industrial filtration manufacturers in Ontario.”
- One direct question about the company: what it makes, who runs it and where it is based.
- The question set is frozen and versioned. The current set is QUALIFY_PANEL_V2, in use since 2026-09-24. A changed question gets a new version, so reports built on different sets are never compared as if they were the same measurement.
The assistants
- Starter audits: ChatGPT with web search on, an OpenAI model with web search off, and Perplexity with web search on, through its Agent API. Perplexity supported its Sonar API until 2026-09-27; audits before that date used Sonar.
- Full audits add Google AI Overviews, Google AI Mode, Gemini and, on request, Microsoft Copilot.
- Each report names the assistants it used, the date of every answer and the model each one reported. Where a provider does not report the model that answered, the report says so.
How we count
- A company counts as named when an answer mentions it as a company or cites its website.
- Names that are also everyday words, places or trade phrases count only when the answer cites the website or clearly presents the name as a company.
- “Named in 4 of 9 answers” means exactly that: four recorded answers out of nine. We report counts rather than a single ranking, because answers change between runs.
- A statement counts as wrong only when the recorded answer contradicts the company’s own details, for example a different city or a website address that is not its own. Starter audits check two details, location and website; they do not fact-check anything else the assistants say.
The evidence we keep
- Every answer used in a report is stored in full, with the assistant, the question, the time and a sha256 fingerprint.
- Each report ships with that evidence ledger, and its machine-readable description (schema.org) points to this page.
- Automatic counts are checked by a person against the recorded answers before a report is sent.
How to check any Arbling report
- Open ChatGPT with web search on, or Perplexity.
- Paste a question from the report, word for word.
- Compare which companies it names with what the report recorded. Your answer may differ in detail; the pattern across several runs is what the report measures.
Limits
- An audit is a snapshot of answers on the dates shown, not a guarantee of future answers or rankings.
- Answers can differ by account, location and time.
- A starter audit does not measure Google search positions.
- We cannot make an assistant name a company. We can show what it reads, and whether that changes, by asking the same questions again after a fix.
What we never do
- Hide text or instructions for AI readers in a report or on a client’s site.
- Publish numbers we did not measure.
- Promise a ranking or a revenue figure.
Questions about a report
Write to hi@arbling.com with the report date and the company name, and we will show you the recorded answer behind any number in it.
