Content signals: what ai-input=no asks of an app reading live pages

Your app reads a public page because a user asked a question, then hands that page to a model. robots.txt has long answered one question: may this agent fetch this path? Content signals answer a second one: what may be done with the content once it is fetched? For an app that answers from live pages, the signal that matters is ai-input. This post explains what it asks of you and where the check belongs. Docs were checked on 2026-11-08. This is general information, not legal advice.

Three signals, three uses

Cloudflare published the Content Signals Policy on September 24, 2025, under CC0, so anyone can implement it. A site states its preferences as comma-separated yes or no values on a Content-Signal line inside a robots.txt group:

User-Agent: *
Content-Signal: search=yes, ai-train=no
Allow: /

Each value covers one use:

• search: building a search index and returning results such as links and short excerpts. The policy explicitly leaves out AI-generated search summaries.
• ai-input: feeding content into one or more AI models, for retrieval-augmented generation, grounding or real-time retrieval for generative AI answers.
• ai-train: training or fine-tuning AI models.

An app that fetches a page at question time and puts it in the prompt is doing exactly what ai-input describes. The Cloudflare authors' Internet-Draft (draft-romm-aipref-contentsignals-00, October 1, 2025) uses the same wording and places AI Input inside the broader "Automated Processing" category of the IETF AI Preferences vocabulary.

yes, no and absent

The decision has three branches, and the third is the one apps get wrong.

• ai-input=yes: the owner allows the use. Read the page and pass it on.
• ai-input=no: the owner asks you not to feed the page into a model. Do not put it in the prompt.
• No ai-input value: the policy says the absence of a signal conveys no meaning. The owner neither grants nor restricts the use, so your own default decides.

Absent is common. Cloudflare says its managed robots.txt runs on over 3.8 million domains, and for those sites it writes search=yes, ai-train=no with no ai-input value, because it does not know what those customers want for that use. So ai-train=no on a page tells you nothing about ai-input, and search=yes does not cover an AI summary either.

The use= extension Cloudflare is testing

Cloudflare's managed robots.txt docs (last updated November 2, 2026) describe content-use, an optional extension that Cloudflare is testing. The managed file now adds use=reference:

User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /

The docs list three values, from least to most permissive: use=immediate (interact, but store and reuse nothing), use=reference (index, excerpt and link back) and use=full (summarize and reproduce). The extension is still a test, so treat it as a hint for now. For an answering app, use=immediate is a reason to keep no stored copy of the page, and use=reference is a reason to quote short passages and link the source.

Where the check belongs

The signal applies to what happens after the fetch, so check it before the page enters the prompt, not after the model has answered. Start with this default, then change it only where your own policy says so:

1. Read robots.txt for the host and find the group for your fetcher's token, or the * group if yours has none.
2. If ai-input=no, drop the page from the context and tell the user the site asked not to be used as AI input.
3. If ai-input is absent, apply your stated default, and say in your docs what it is.
4. Store the signal you read next to the page, together with the fetch time.

The robots.txt read itself, with its own agent tokens and caching rule, is covered in robots.txt and AI fetchers. This check adds one line to the same read.

A few lines of parsing read the Content-Signal line for your token directly:

import urllib.request
import urllib.error


def content_signals(host, token):
    try:
        with urllib.request.urlopen(f"https://{host}/robots.txt", timeout=10) as r:
            text = r.read(500 * 1024).decode("utf-8", "replace")
    except urllib.error.URLError:
        return {}
    groups, agents, in_rules = [], [], False
    for raw in text.splitlines():
        line = raw.split("#", 1)[0].strip()
        if ":" not in line:
            continue
        key, value = (p.strip() for p in line.split(":", 1))
        key = key.lower()
        if key == "user-agent":
            if in_rules or not groups:
                agents, in_rules = [], False
                groups.append((agents, {}))
            agents.append(value.lower())
        elif groups:
            in_rules = True
            if key == "content-signal":
                for pair in value.split(","):
                    if "=" in pair:
                        k, v = (p.strip().lower() for p in pair.split("=", 1))
                        groups[-1][1][k] = v
    for wanted in (token.lower(), "*"):
        for names, signals in groups:
            if wanted in names:
                return signals
    return {}


signals = content_signals("example.com", "ExampleAppFetcher")
print(signals.get("ai-input", "absent"))

An empty result means no signal was read, which is the "absent" branch, not a "yes".

What the signal does not do

Cloudflare is plain about the limits: content signals state preferences and are not technical countermeasures, and some companies may ignore them. The policy text still frames them as a condition of access, and its last paragraph says restrictions expressed through the signals are reservations of rights under Article 4 of EU Directive 2019/790. What that means for your app depends on where you operate. This is not legal advice.

The signal also sits on top of the rest of the boundary, never in place of it. Check the site's terms of service and your provider's acceptable-use policy (AUP) as well. Logins, paywalls and private APIs stay out of scope whatever a robots.txt file says. The wider "public pages only" rule is laid out in why LLM apps need live web data.

Reading the page after the check

Once the check passes, the read itself should be current, because the signal you just read and the page you pass on should come from the same moment. Live Web Data API keeps that rule simple. Every request fetches the page live; there is no cache. The page comes back as Markdown, plain text or HTML, and statuses[].fetchedAt records when it was read, so the signal and the fetch time can be stored side by side. The request fields are in the API docs.