robots.txt and AI fetchers: what a user-triggered read signals

Your app reads a public page because a user pasted a URL or asked a question. Is that a crawl that must follow robots.txt, or a visit on the user's behalf? The three largest AI vendors answer that question in three different ways. This post lays out what each one documents, what the standard says, and the policy your own fetcher needs. Docs were checked on 2026-11-07. This is general information, not legal advice.

Three jobs, three kinds of agent

Every vendor now splits its traffic into separate agents, each with its own robots.txt token:

• Training: GPTBot (OpenAI) and ClaudeBot (Anthropic). Google uses the Google-Extended token, which covers Gemini training and grounding but has no HTTP user agent string of its own, and does not affect inclusion or ranking in Google Search.
• Search: OAI-SearchBot (OpenAI) and Claude-SearchBot (Anthropic) feed the search features behind answers.
• User-triggered reads: ChatGPT-User (OpenAI), Claude-User (Anthropic) and Google's user-triggered fetchers, such as Google-Agent, which acts on user requests in agents running on Google infrastructure.

OpenAI notes that each robots.txt setting is independent: a site can allow OAI-SearchBot and disallow GPTBot. It also says a robots.txt change can take about 24 hours to reach its search results.

The split matters because a site owner who blocks the training agent has said nothing yet about reads a user asked for.

Where the vendors differ on user-triggered reads

For training, all three vendors document a robots.txt token a site can use to opt out. The user-triggered agents are where they part ways:

OpenAI adds that ChatGPT-User does not crawl the web automatically and does not decide what appears in its search. Google says its list of user-triggered fetchers is not exhaustive and that user agent strings can be spoofed. Anthropic publishes the IP addresses of its crawlers and advises against IP blocking, because it can stop the bots from reading robots.txt at all.

So the same Disallow: / line under a user-triggered token means "do not read" to one vendor and a preference that may not apply to another.

What the standard says

RFC 9309, the Robots Exclusion Protocol, settles less than people expect. Its introduction states that the rules are not a form of access authorization, and it calls the protocol no substitute for real content security. It describes how a crawler should behave, not who counts as a crawler.

A few of its rules are worth copying into any fetcher that does read robots.txt:

• Match the group for your product token without regard to case, and keep the token to letters, underscores and hyphens.
• Do not use a cached robots.txt for more than 24 hours, unless the file cannot be reached.
• A 4xx response for robots.txt means the file is unavailable, and the crawler may access any resource.
• A 5xx response means it is unreachable, and the crawler must assume a complete disallow.
• Parse at least the first 500 kibibytes of the file.

The policy your own app needs

An app that reads live pages on request sits in the same place as ChatGPT-User or Claude-User. You have to pick a rule, and the vendors' docs show that "a user asked" is a choice, not an exemption. Start with the stricter default, then relax it only where you can justify it:

1. Name your fetcher. Send a user agent with a stable product token, so site owners can write a group for it.
2. Read robots.txt for that token before the first fetch from a host, and keep the copy for at most 24 hours.
3. Honor a disallow for user-triggered reads too. Tell the user the site asked not to be read by your agent, instead of failing silently.
4. Treat a 5xx on robots.txt as a full disallow, as RFC 9309 requires.
5. Check the terms of service and your provider's acceptable-use policy (AUP). robots.txt does not replace either, and neither covers logins, paywalls or private APIs, which are out of scope for live web data.

A site that wants to stop user-triggered agents but keep search would write separate groups. On example.com it looks like this:

User-agent: Claude-User
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: OAI-SearchBot
Allow: /

If your app checks its own token in Python, the standard library is enough:

from urllib.robotparser import RobotFileParser

rp = RobotFileParser("https://example.com/robots.txt")
rp.read()
print(rp.can_fetch("ExampleAppFetcher", "https://example.com/docs/page"))

Run this once per host and keep the result for at most 24 hours. Pass the decision to the user, and record when you checked it, just as you record the fetch time of the page itself.

Where this fits

robots.txt is the first gate of the "public pages only" boundary described in why LLM apps need live web data. It is a statement of the owner's wishes, read per host and per token. If you run the same check inside a scheduled job, agentscraper.cc shows it as a robots.txt check step.

Vendor docs on this topic change often. Re-check the three vendors' pages before you copy their rules into your own policy. This post is not legal advice.