Six myths about AI agents and your website
Common beliefs, the published facts that correct them, and what to do instead.
Am I Ready for Agents? editors · 25 September 2026 · Reviewed September 2026
Most myths about AI agents come from treating all AI visitors as one thing, or a file or a score as a guarantee. robots.txt is a request, refusing training does not have to mean leaving search, llms.txt is optional, a scanner score is not a task test, a bot's name proves nothing, and most businesses do not need a new payment method yet.
Myth 1: does robots.txt block AI bots?
Fact: robots.txt asks. RFC 9309, the standard behind it, says its rules are not a form of access authorization. Well-behaved crawlers follow them; others may not, and Vercel's verified bots directory notes that Meta-ExternalFetcher, because a user started the fetch, may bypass robots.txt rules.
What to do: make sure your bot protection at the CDN or host enforces the same position your robots.txt states. See robots.txt and bot rules.
Myth 2: does refusing AI training mean leaving AI search?
Fact: operators increasingly separate the two. Google says blocking Google-Extended does not affect inclusion or ranking in Google Search, and Cloudflare reports the same split for Applebot-Extended. OpenAI runs GPTBot for training and OAI-SearchBot for ChatGPT search, which the Vercel directory describes as not crawling for model training. In September 2026 Cloudflare added a setting that refuses training while keeping a site discoverable in search.
What to do: decide per kind of visitor. The memo on which AI bots to let in sets out three positions.
Myth 3: does every site need an llms.txt file now?
Fact: llms.txt is a proposal, revised to version 2 in August 2026. It helps agents that look for it and does not control crawling. It is useful when you want to point language models to the pages that matter, and it is not required by any standard we have read.
What to do: decide yes or not yet, and name who would keep the file current. Either answer counts on the self-check. Our llms.txt briefing goes further.
Myth 4: does a high scanner score mean agents will succeed?
Fact: scanners check lists of files, settings and page features, each on its own scale. ora.ai made the distinction itself when it introduced Deep Scan: a checklist confirms a file exists; a benchmark confirms the system works. A score summarises checks. It does not by itself show that customers' agents complete tasks on your site.
What to do: read the failed checks that sit on your main task, and test the task itself. See how to read a scanner report.
Myth 5: if a bot says it is GPTBot, is it GPTBot?
Fact: a user agent name is text any script can copy. Platforms verify bots by published IP ranges, reverse DNS or Web Bot Auth signatures, which are built on HTTP Message Signatures (RFC 9421). Since 28 August 2026 Cloudflare validates Web Bot Auth signatures automatically when operators submit bots to its directory.
What to do: ask whether your bot protection uses a verified bot list, and let verified agents through at checkout or booking. See verifying agents and bots.
Myth 6: does being agent-ready mean adopting a new payment method?
Fact: agent payment approaches exist, but none is universal. x402 uses the 402 Payment Required status code and added batch settlement in May 2026, and Cloudflare's Is It Agent Ready scanner also checks for ACP, UCP and MPP. For most stores the nearer questions are whether the full price is readable before the last step and whether fraud rules decline a legitimate customer's agent.
What to do: settle the price and fraud questions first. See agent payments and fraud rules.
Why do these myths persist?
Because the subject moved quickly in 2026, and most coverage treats AI visitors as one group. The news page lists the dated announcements behind the facts above. Take the self-check to see which of these beliefs your own answers rely on.