AI agents and your website: common questions

Software that carries out a task on the web for a person, such as finding a product, comparing prices, filling in a form or paying. See what agents do.

A crawler collects pages in bulk on its own schedule, to train a model or build an index. An agent works on one task for one person, at that person's request. Blocking a crawler changes what an AI system knows about you later; blocking an agent stops a customer's request now. See what agents do.

It depends on the bot. Training crawlers, search crawlers and agents acting for a user do different jobs, and several operators let you refuse one without refusing the others. Google, for example, says blocking Google-Extended does not affect inclusion or ranking in Google Search. See which AI bots to let in.

Increasingly, yes. Google says blocking Google-Extended does not affect inclusion or ranking in Google Search, Cloudflare reports the same split for Applebot-Extended, and in September 2026 Cloudflare added a setting that refuses training while keeping a site discoverable in search. See robots.txt and bot rules.

It asks them to stay out. The robots.txt standard, RFC 9309, says its rules are not a form of access authorization. Enforcement needs bot management at your CDN or host.

By verification, not by name. Platforms check that requests come from the operator's published IP ranges, use reverse DNS, or check Web Bot Auth signatures, which are built on RFC 9421. See the lesson on verifying agents and bots.

It is optional. llms.txt is a proposal for a short markdown guide to your site written for language models, revised to version 2 in August 2026. It helps agents that look for it and does not control crawling.

Three ways for agents to act directly rather than click through pages. MCP connects AI applications to tools and data, WebMCP is a proposal for web pages to offer tools to agents in the browser, and A2A lets agents from different organisations hand each other tasks. See forms, sign-in and consent.

No. It scores your own answers in your browser. For an automated check of public pages, see tools that check.

Nothing leaves the page. The self-check keeps answers only in the open browser tab, with no storage, no submission and no tracking. Reloading clears them.

Yes scores 2 points, not sure 1 and no 0. Each area becomes a percentage, the five areas count equally, and the average places you in one of five levels, capped at one above your weakest area. The full rubric is on readiness levels.

Because of the weakest-area rule. Your level can be at most one above the level your lowest area would get on its own, since an agent's task fails if any step fails. See how to read your self-check result.

It takes your answer counts per area, shows the reasoning step by step, offers a view weighted by your own priorities and finds the fewest answer changes that would move you up a level. See the readiness calculator.

No. Scanners test public pages on their own scales; ora.ai, for example, grades from A+ to F. Our levels score your own answers with an editorial rubric, so they do not map onto any scanner's grade.

The people who own the pieces: marketing or ecommerce, whoever runs the website platform, whoever manages the CDN or bot protection, and payments or fraud if you sell or take bookings online. Comparing answers is the point. See how to run a 30-minute review.

We do not rank tools. Tools that check lists scanners and services and what each one looks at, so you can pick by the question you need answered.

Yes, by what each checks, not by score. The side-by-side view puts two to five entries next to each other, field by field, from their own pages.

Few publish prices. Cloudflare states that AI Crawl Control is available on all plans and that Markdown for Agents is included at no cost on Pro, Business and Enterprise plans and SSL for SaaS. The other tools on our list do not state a price on the pages we read. See tools that check.

Not as far as the scanners' own pages say. ora.ai states that it scans public URLs only and cannot scan login-gated content; the other scanners on our list do not state their limits.

No. The first steps are decisions and settings: your position on AI bots, prices and policies readable as text, and a named owner for your agent policy. Your CDN or host may already offer bot controls. See ask your web team.

We do not know, and we do not promise it. What is true is narrower: an agent that cannot read your price or finish your form cannot complete that task for its user. Measure business impact with your own data.

The Am I Ready for Agents? editors. See about this site.

A company that makes agent readiness tools. It describes a Scan that runs real agents against a site, a Journey tool that shows one agent's path on a page, and grades from A+ to F. See its entry on tools that check.

No. The self-check, the rubric and the readiness levels are our editorial work. They are not derived from ora.ai's methodology and do not produce ora.ai scores.

A scanner built by Vercel with ora.ai, which runs its scans. It returns a score with evidence and recommendations, split into essential, recommended and bonus checks. We describe it as a partnership, and ora.ai's announcement is listed on our news page.