Web data budget for AI apps: fetch cost plus token cost

Every page that enters a model's context is paid for twice: once to fetch it and once as input tokens. Most cost estimates count only the first. This guide gives you a worksheet that counts both, then covers the levers that move each one.

The two bills

The fetch bill is what your web data provider charges per page. With Live Web Data API, every successful read is a live fetch and is billed; how many credits a page takes depends on how it had to be fetched (a standard page, a page behind bot protection, or a browser-rendered page), and a failed one costs 0. Current plans and credit amounts are on the pricing page.

The token bill is what your model provider charges for the page's content as input. It scales with page length and format. A page passed as raw HTML carries scripts, attributes and navigation; the same page as main-content Markdown carries mostly the words and structure that matter.

The worksheet

Fill in your own numbers. Label every assumption so you can replace it with a measurement later.

Two notes. Live Web Data API has no cache, so the only way to bill fewer fetches is to read fewer pages live: store copies on your side and reuse them within a max age. If your provider caches and bills cache hits differently, add a line for it from its billing docs. And if a fetch fails, you pay neither bill for that page, but your app still has to answer.

Measure tokens per page yourself

Do not guess line G. Take ten representative URLs, fetch each as Markdown with your usual settings, and count tokens with your model's tokenizer: tiktoken for OpenAI models, the token counting endpoint for Anthropic models. Then fetch the same pages as HTML and count again. The difference is what your format choice is worth, and the guide on Markdown, text or HTML for LLMs explains the trade-offs.

Lever 1: truncate long pages

Long pages are where the token bill grows (the Markdown, text or HTML guide covers the format side). text.maxCharacters truncates Markdown and text to a limit you set; the default is 100,000 characters, and results[].textTruncated says when a page was cut. The post on the token cost of web pages shows how to turn a token ceiling into that limit. Choose the limit by page type:

• Product, pricing and status pages: the answer is usually near the top, so a few thousand characters is often enough.
• Docs and changelogs: keep more, or fetch the specific section URL instead of the whole page.
• Long articles: truncate for classification, keep more for summaries.

Test the limit on your own pages: check that the answers you need survive the cut.

Lever 2: store copies and set a max age

A max age on copies you store stops the same page from being fetched live again and again within minutes. That makes answers faster, keeps your load on the target site proportionate and, because Live Web Data API bills every live read and has no cache of its own, cuts line E directly. Keep the fetchedAt time with each stored copy so you know its age. The guide on live vs cached web data has starting defaults per page type.

Lever 3: send fewer pages

The cheapest page is the one you do not send. Only include pages the question needs, and drop duplicates that redirect to the same final URL (the response includes results[].url, the final URL after redirects).

Failed requests

Failures are normal on the open web: pages move, sites time out, some content needs JavaScript. How they are billed varies by provider, so read the billing docs. In Live Web Data API, timeouts, not-found pages, restricted targets and pages without usable content all cost 0 credits, and each comes back with an error tag such as CRAWL_NOT_FOUND or CRAWL_LIVECRAWL_TIMEOUT. That keeps a retry cheap, but retries still add latency, so cap them.

Batching and rate limits

Up to 10 URLs fit in one request. Batching saves round trips when you know several URLs up front; each page is still billed on its own. A URL that fails to fetch does not fail the others; invalid parameters or too few credits fail the whole request.

Rate limits are set per plan and are listed on the pricing page. Each URL in a batch counts toward the per-second limit, so a 10-URL request uses ten units, not one. Responses carry X-RateLimit-Limit and X-RateLimit-Remaining headers, and a 429 comes with Retry-After. If your peak is bursty, put a queue in front of the fetch so a spike waits instead of failing.

Where Live Web Data API fits

The product's billing is simple to model: every successful read is live and billed by how the page had to be fetched, and failures cost 0. Sign-up is for businesses only. Only Business, Enterprise and Custom sign-ups are reviewed before checkout, usually within 1 business day. See current plans and plug them into line F.

Checklist

• Both bills are in the budget: fetch and tokens.
• Tokens per page are measured on your sources, in your format.
• Each page type has a character limit and a max age.
• Failure billing is confirmed in your provider's docs.
• Rate limits and peaks are planned with a queue and retries capped.
• Assumptions are labelled and replaced with measurements after launch.