AI agents and your website: common questions

Software that carries out a task on the web for a person, such as finding a product, comparing prices, filling in a form or paying. See what agents do.

A crawler collects pages in bulk on its own schedule, to train a model or build an index. An agent works on one task for one person, at that person's request. Blocking a crawler changes what an AI system knows about you later; blocking an agent stops a customer's request now. See what agents do.

It depends on the bot. Training crawlers, search crawlers and agents acting for a user do different jobs, and several operators let you refuse one without refusing the others. Google, for example, says blocking Google-Extended does not affect inclusion or ranking in Google Search. See which AI bots to let in.

Increasingly, yes. Google says blocking Google-Extended does not affect inclusion or ranking in Google Search, Cloudflare reports the same split for Applebot-Extended, and in September 2026 Cloudflare added a setting that refuses training while keeping a site discoverable in search. See robots.txt and bot rules.

It asks them to stay out. The robots.txt standard, RFC 9309, says its rules are not a form of access authorization. Enforcement needs bot management at your CDN or host.

By verification, not by name. Platforms check that requests come from the operator's published IP ranges, use reverse DNS, or check Web Bot Auth signatures, which are built on RFC 9421. See the lesson on verifying agents and bots.

It is optional. llms.txt is a proposal for a short markdown guide to your site written for language models, revised to version 2 in August 2026. It helps agents that look for it and does not control crawling.

Three ways for agents to act directly rather than click through pages. MCP connects AI applications to tools and data, WebMCP is a proposal for web pages to offer tools to agents in the browser, and A2A lets agents from different organisations hand each other tasks. See forms, sign-in and consent.

It scores your own answers in your browser. For an automated check of public pages, see tools that check.

Nothing leaves the page. The self-check keeps answers only in the open browser tab, with no storage, no submission and no tracking. Reloading clears them.

Yes scores 2 points, not sure 1 and no 0. Each area becomes a percentage, the five areas count equally, and the average places you in one of five levels, capped at one above your weakest area. The full rubric is on readiness levels.

Because of the weakest-area rule. Your level can be at most one above the level your lowest area would get on its own, since an agent's task fails if any step fails. See how to read your self-check result.

It takes your answer counts per area, shows the reasoning step by step, offers a view weighted by your own priorities and finds the fewest answer changes that would move you up a level. See the readiness calculator.

No. Scanners test public pages on their own scales; ora.ai, for example, grades from A+ to F. Our levels score your own answers with an editorial rubric, so they do not map onto any scanner's grade.

The people who own the pieces: marketing or ecommerce, whoever runs the website platform, whoever manages the CDN or bot protection, and payments or fraud if you sell or take bookings online. Comparing answers is the point. See how to run a 30-minute review.

No. DataDome verifies, allows, blocks or charges bots and AI agents to stop fraud; its pages read do not describe testing whether an agent can complete a task on your site. The two work together: DataDome controls who gets in, ora.ai checks whether the agent that gets in can finish the job.

AEO (answer engine optimization) platforms get you surfaced: they track and improve how your brand shows up in AI answers, the find step. Agent experience gets you converted: a customer's agent that arrives can read your site, get in and finish a booking, a sign-up or a purchase. ora.ai is an agent experience platform covering discovery, access, usability and payments. See AEO vs agent experience.

No. Scrunch focuses on how a brand appears in AI answers and serves AI-optimized pages to agents; its pages read do not describe testing whether an agent can complete a task. ora.ai checks discovery, access, usability and payments, and shows an agent's path through a page.

Tools that check lists scanners and services and what each one looks at, so you can pick by the question you need answered.

Yes. The side-by-side view puts two to five entries next to each other, field by field, from their own pages.

Scrunch, 514.ax and Profound publish plan prices, while TollBit offers publishers a free platform without a public price list. Cloudflare states that AI Crawl Control is available on all plans and that Markdown for Agents is included at no cost on eligible paid plans. See the entries for prices and their cited sources. See tools that check.

Not as far as the scanners' own pages say. ora.ai states that it scans public URLs only and cannot scan login-gated content; the other scanners on our list do not state their limits.

No. The first steps are decisions and settings: your position on AI bots, prices and policies readable as text, and a named owner for your agent policy. Your CDN or host may already offer bot controls. See ask your web team.

An agent that cannot read your price or finish your form cannot complete that task for its user. Measure business impact with your own data.

Pages draw on public standards documents, crawler directories and vendor announcements, with links to the sources. See about this site.

A company that makes agent readiness tools. It describes a Scan that runs real agents against a site, a Journey tool that shows one agent's path on a page, and grades from A+ to F. See its entry on tools that check.

The readiness levels page explains the points, level bands and weakest-area rule used by the self-check.

A scanner built by Vercel with ora.ai, which runs its scans. It returns a score with evidence and recommendations, split into essential, recommended and bonus checks. We describe it as a partnership, and ora.ai's announcement is listed on our news page.