Live vs cached web data: where the cache belongs in LLM apps

Every copy of a web page has an age. A live fetch gives you age zero at the cost of waiting for the site. A cached copy is fast but might be hours or days old. Good LLM apps decide this per page type, keep the decision in configuration and send the age along with the content. This guide explains how vendor max-age controls work, when a cache on your own side pays off, and how to tell the model how old its source is.

How max age works at providers that cache

Many contents APIs follow a pattern borrowed from HTTP caching (RFC 9111): you state the oldest copy you will accept.

• If the provider's cache holds a copy younger than your limit, you get it.
• If not, the page is fetched live and the cache is updated.
• A limit of zero means always live.
• Some APIs also offer cache only: never fetch, return what is stored or nothing.

Units differ between vendors. Exa uses hours (maxAgeHours), Firecrawl uses milliseconds (maxAge; 3600000 is one hour), and other APIs use seconds. Vendor parameters were checked against their public docs in October 2026; they change, so check again before you rely on them. When you compare providers or move between them, convert first: a max age of 24 means a day in one API and 24 milliseconds in another.

This site is published by Teneo Labs, which makes Live Web Data API, the third column in the table above. That API keeps no cache: every request reads the page live. It accepts maxAgeHours: 0 so clients written for other providers keep working; any other value is rejected with a 400 instead of silently returning something older than you asked for.

When a cache on your side pays off

With a live-only API, the cache moves into your app. That sounds like more work, but it puts the freshness decision where the knowledge is: you know which sources change hourly and which change once a quarter.

A small store is enough. Key it by URL and country, because a page read from Germany and the same page read from the US can differ; the guide on web pages by country explains why. Store the content, the fetchedAt time from the response and the HTTP status. Before each read, compare the stored fetch time with the max age for that source type and only fetch live when the copy is too old.

These are starting points, not measurements. Two rules make the table work in production. First, put the max age in configuration per source, not in code. Second, log how often you serve a stored copy and how often you fetch live, so you can see whether the numbers fit. Every live fetch is a billed request, so a stored copy that is still young enough is also the cheapest read; the web data budget guide covers that side.

Latency: measure, do not assume

A read from your own store returns as fast as your database. A live fetch takes as long as the target site plus extraction. The difference depends on the site, the country and the time of day, so published averages tell you little about your pages.

Measure it yourself: fetch a sample of your real sources live and look at searchTime in each response. That gives you numbers to design timeouts and user-facing waits around.

Timeouts and fallback to a stored copy

A live fetch can hang. livecrawlTimeout caps the wait in milliseconds; the default is 10,000 and the maximum is 90,000. When the limit is reached, the result is the error CRAWL_LIVECRAWL_TIMEOUT, and failed fetches cost 0 credits.

If you keep stored copies, you have a middle path: when the live fetch fails, serve the last stored copy and mark it as older. That is often the right answer for a user waiting on a response, but it must be visible:

• Show the fetch time next to the answer ("page read 2 days ago").
• For high-stakes pages such as prices, treat a failed live fetch as an error and say the current page could not be read.

Record the fetch time of every page

Each URL in a request gets an entry in statuses[]. On success, source is livecrawled and fetchedAt holds the time the page was read during this request. Store both next to the content, together with the country. It is the difference between "we think this is current" and "we know when this was read".

Tell the model how old the page is

The model cannot see HTTP headers or your store. If freshness matters to the answer, put it in the prompt:

<source url="https://example.com/pricing" fetched="2026-10-27T09:14:00Z" country="us">
...page markdown...
</source>
Answer from the source above. If the question depends on information newer than the fetch time, say so.

Use the fetchedAt value from the response, or the time you stored the copy. Do not rely on the page's Last-Modified header: dynamically generated pages often report the current time, and some static pages never update it. RFC 9111 treats such dates as heuristics for exactly this reason.

Where Live Web Data API fits

The API is the live half of this design: each request reads the page now, returns statuses[].fetchedAt, and caps the wait with livecrawlTimeout. Failed fetches, including timeouts, cost 0 credits. You can try live reads on your own URLs in the playground once your plan is active, or read the response fields in the API reference.

Checklist

• Every source type has a max age, stored in configuration.
• Stored copies are keyed by URL and country and keep their fetch time.
• Units are converted when comparing or switching providers.
• livecrawlTimeout matches what your users can wait for.
• A stored copy served after a failed live fetch is shown with its age, or treated as an error, by page type.
• The fetch time and country travel with the content into the prompt.