Arbling's scanner

A scan started on arbling.com sends its requests under one of two user agents:

• ArblingAIReadinessBot/1.0 (+https://arbling.com/bot)
• ArblingMysteryShopper/1.0 (+https://arbling.com/mystery-shopper)

Both are Arbling's scanner, and both lead to this page.

Only when a person asks: someone enters a website's address on arbling.com, or a customer scans their own site from their account. Nothing rescans a website's public pages on a schedule. A store connected to Arbling is re-read once a day through its platform's API, with the owner's key. A public scan on arbling.com: at most 12 an hour from one address.

• the home page and up to three pages it links to;
• /robots.txt, /sitemap.xml and one sitemap it lists;
• /llms.txt;
• the product list a store publishes: Shopify's /products.json or WooCommerce's Store API;
• a sample of product pages: up to twelve, and up to four more for the checks that act like an agent;
• the files AI agents look for: /.well-known/ucp, /.well-known/mcp.json, /.well-known/mcp/server-card.json, /.well-known/agents.json, /.well-known/acp.json, /.well-known/http-message-signatures-directory and /agents.md;
• the cart, to see whether an agent can reach it: /cart on any store, and /cart.js on Shopify;
• in a customer's photo audit, up to five product pictures, each checked with a request that reads none of the picture, or one byte of it.

A customer's scan of their own catalogue reads up to 60 product pages from the sitemap, up to three collection pages, and, only when nothing else works, up to 60 pages of the site.

A scan sends at most 12 requests every two seconds to one website, and drops a request that takes longer than 8 seconds, 25 for a connected store's API and the catalogue scan's last resort. An answer of "too many requests" or "forbidden" is tried once more after two seconds, and not again. The paid browser audit loads a page as a browser does.

The scanner reads a site's robots.txt to report on it, not to follow it. A scan runs because a person asked for it, and its result shows which AI crawlers and agents the site's rules would turn away.

Write to hi@arbling.com with your website's address. A site can also refuse the scanner at its server or its CDN: it must refuse both user agents, because a scan sends some of its requests under each. The scanner has no fixed IP address: it runs from Arbling's own servers, with no proxy and no scraping service, so refuse it by its names.