Why LLM apps need live web data, and when they do not

A language model answers from two places: what it learned in training and what you put in its context window. The first is frozen at a cutoff date. The second is the only way a model can know what a public web page says today. This guide explains when that matters, what "real-time" should mean in numbers, and which tool fits which job.

What goes stale after the knowledge cutoff

Not every fact ages at the same speed. A practical way to sort the pages your app depends on:

The dangerous part is not that the model lacks the fact. It is that the model rarely says so. Asked about a page it has not seen in its current form, it reconstructs a plausible version from older training data and states it with the same confidence as everything else.

The fix is simple to describe: fetch the current page, put its clean content into the prompt, and tell the model where it came from and when it was read.

What "real-time" should mean in numbers

"Real-time web data" is a phrase without a unit. In practice, every API that offers it is making one of three promises:

• Live fetch: the page is requested from the site now. It is as fresh as it gets and as slow as the site.
• Cache within a max age: a stored copy is returned if it is younger than a limit you set; otherwise the page is fetched live.
• Cache only: a stored copy of any age, or nothing.

Vendors express the limit in different units. Exa's contents endpoint uses maxAgeHours, Firecrawl's maxAge is in milliseconds, and others use seconds. The idea is the same: freshness is a number you choose per page type. The guide live vs cached web data has starting defaults for each type.

Ask any provider, including us, three questions: How old can the content be? How do I see when this result was read? What happens when the live fetch fails? Our own answers: every read is live, statuses[].fetchedAt gives the time, and a failure returns an error tag at 0 credits.

Contents API vs search API

Two different jobs often hide behind "give the model web access":

1. Finding pages. You have a question, not a URL. A search API returns ranked URLs and snippets.
2. Reading pages. You have the URL. A contents API returns the page's content, cleaned for a model.

Many apps only need the second. A docs assistant knows its docs URLs. A pricing checker knows the pricing pages. A changelog summarizer knows where the changelog lives. Using a search engine for these adds cost and a ranking step that can return the wrong page.

If you need both, run them in order: search to pick URLs, then a contents call to read them. Do not expect one to replace the other.

Public pages only

Live web data, as this site uses the term, means public pages: anything a visitor can read without logging in, paying or using a private API. That boundary is not a technicality. It is where the legal and ethical questions start, and the product's acceptable-use policy draws it in the same place.

Two habits help:

• Read the site's robots.txt (RFC 9309 describes the format) and its terms of service before you fetch from it at volume. robots.txt is a signal of the site owner's wishes, not an access control, and respecting it is part of being a good citizen on the web.
• Keep volumes proportionate. Fetching a page once an hour for a feature is different from hammering it.

This is general information, not legal advice.

When a live fetch is the wrong tool

A live fetch is not always the answer:

• The fact does not change. Training data or a static reference is fine.
• You need hundreds of related pages at once. That is crawling or retrieval over an index, and the questions become chunking and refresh schedules. Our sister site has a guide on web content ingestion for RAG that covers what happens after the fetch.
• The content only appears after JavaScript runs. A static fetch returns little or nothing. Live Web Data API flags these pages (a THIN_CONTENT warning or a CRAWL_NOT_RENDERED error, which is not charged) and can read them in a browser when you set render: true.
• You need structured fields extracted into a schema. That is a separate step after reading the page.

Build vs buy

A minimal fetcher is an HTTP GET and an HTML-to-Markdown library. A production one also needs:

• timeouts and retries that do not stall your users,
• main-content extraction, and a choice of format per task (see Markdown, text or HTML for LLMs),
• a store of fetched copies keyed by URL and country, with a max age per source type,
• connections in each country whose version of a page you need (see web pages by country),
• a clear error for each failure (not found, timed out, not rendered, unsupported content type),
• billing or quotas you can predict (see the web data budget guide).

If that list is your product, build it. If it is plumbing under your product, buying it is usually cheaper than running it.

Where Live Web Data API fits

Live Web Data API is the contents step: you send up to 10 public URLs per request and get each page back as Markdown, text or HTML. Every request reads the page live (there is no cache), countryCode sets where the page is read from, statuses[].fetchedAt records when each page was read, and render reads JavaScript-only pages in a browser. Failed or blocked requests cost 0 credits. It does not search, crawl or extract structured data. Sign-up is for businesses only. Only Business, Enterprise and Custom sign-ups are reviewed before checkout, usually within 1 business day. Read the docs or request access.

Checklist

• List the pages your app depends on and sort them by how fast they change.
• Decide per type: training data, a stored copy within a max age, or live.
• Confirm you know the URLs; add a search step only if you do not.
• Check that every target is public and that robots.txt and the terms allow your use.
• Pass the URL and fetch time into the prompt with the content.
• Decide what the app shows when a fetch fails.