Arbling's scanner
Arbling's scanner reads a limited set of a website's public addresses, the way an AI shopping agent acting for a person would. It runs only when a person asks for a scan.
Its two names
A scan started on arbling.com sends its requests under one of two user agents:
ArblingAIReadinessBot/1.0 (+https://arbling.com/bot)ArblingMysteryShopper/1.0 (+https://arbling.com/mystery-shopper)
Both are Arbling's scanner, and both lead to this page.
When it runs
Only when a person asks: someone enters a website's address on arbling.com, or a customer scans their own site from their account. Nothing rescans a website's public pages on a schedule. A store connected to Arbling is re-read once a day through its platform's API, with the owner's key. A public scan on arbling.com: at most 12 an hour from one address.
Arbling's audit reader, ArblingAuditBot, runs when Arbling prepares an AI-visibility audit of a company. It has its own section below.
What it reads
- the home page and up to three pages it links to;
/robots.txt,/sitemap.xmland one sitemap it lists;/llms.txt;- the product list a store publishes: Shopify's
/products.jsonor WooCommerce's Store API; - a sample of product pages: up to twelve, and up to four more for the checks that act like an agent;
- the files AI agents look for:
/.well-known/ucp,/.well-known/mcp.json,/.well-known/mcp/server-card.json,/.well-known/agents.json,/.well-known/acp.json,/.well-known/http-message-signatures-directoryand/agents.md; - the cart, to see whether an agent can reach it:
/carton any store, and/cart.json Shopify; - in a customer's photo audit, up to five product pictures. Each is first checked with a request that reads none of the picture, or one byte of it. The pictures that answer are then downloaded, up to 3.5 MB each and within six seconds, and sent to Google's Gemini, the AI model that grades them.
A customer's scan of their own catalogue reads up to 60 product pages from the sitemap, up to three collection pages, and, only when nothing else works, up to 60 pages of the site.
How fast
The requests a scan sends under ArblingAIReadinessBot go at most 12 every two seconds to one website. Its agent checks, sent under ArblingMysteryShopper, are not held to that pace yet. A request is dropped after 8 seconds, after 25 for a connected store's API, and after 22 for the catalogue scan's last resort. When Arbling fills in the details of a customer's own catalogue, it reads their pages under the scanner's name and waits up to 20 seconds. An answer of "too many requests" or "forbidden" is tried once more after two seconds, and not again. The paid browser audit loads a page as a browser does.
robots.txt
The scanner reads a site's robots.txt to report on it, not to follow it. A scan runs because a person asked for it, and its result shows which AI crawlers and agents the site's rules would turn away.
Asking it to stop
Write to hi@arbling.com with your website's address. A site can also refuse the scanner at its server or its CDN: it must refuse both user agents, because a scan sends some of its requests under each. The scanner has no fixed IP address: it runs from Arbling's own servers, with no proxy and no scraping service, so refuse it by its names.
Two more Arbling user agents
ArblingSiteVerify/1.0 (+https://arbling.com)reads the home page of a site a customer adds, to confirm that the customer controls it, including a WordPress site connected with a code.ArblingImagePipeline/1.0downloads product pictures from a customer's own catalogue, up to 16 MB each and within eight seconds, to check their size, type and quality. It changes no picture.
ArblingAuditBot, the audit reader
When Arbling prepares an AI-visibility audit of a company, it reads under this user agent:
ArblingAuditBot/1.0 (+https://arbling.com/bot; hi@arbling.com)
It reads:
- the audited company's own website: at most 20 of its pages (3 in a mini audit), its
/robots.txt, sitemap and/llms.txt, and the site icons its home page links to; - for up to five competitors named in the AI answers, only the home page, the site icons it links to,
/robots.txtand/llms.txt; - Wikipedia and Wikidata, through their public API.
The icons are printed next to each company's name in the report. For an icon it may also ask Google's public favicon service. That request goes to Google, not to the company's site.
On a competitor's site it follows robots.txt and starts at most one request every two seconds. On the audited company's own site it reads robots.txt to report on it, as the scanner does, and sends at most two requests at a time. Every request is a plain GET. It fills in no forms and signs in to no accounts.
The audit's robot shopper is switched off at present. It runs only for a customer's own audit order: it opens that customer's own pages in a browser, the way a buyer would, once per audit. It sends this user agent:
Mozilla/5.0 (compatible; ArblingMysteryShopper/1.0; +https://arbling.com/bot)
It opens at most 25 pages.
It only opens pages. It types nothing and presses no buttons.
To ask it to stop, write to hi@arbling.com. A site read as a competitor can also refuse it in robots.txt with User-agent: ArblingAuditBot and Disallow: /.
Before 10 November 2026, audits read websites under other names: ArblingAudit/hosted (read-only), Mozilla/5.0 (compatible; ArblingReport/1.0; +https://arbling.com), Mozilla/5.0 (compatible; ArblingAudit/1.0; +https://arbling.com), ArblingAudit/technical (read-only) and Mozilla/5.0 (ArblingAudit; read-only). The audit's robot shopper used an ordinary HeadlessChrome browser name.
