# Live Web Data: guides and posts
> Live web data for LLM apps, explained plainly: how fresh a page must be, which country to read it from, which format to pass and what each page costs.
# Why LLM apps need live web data, and when they do not
Source: https://livewebdata.com/guides/why-llm-apps-need-live-web-data
> Why LLMs need live web data: what goes stale after the knowledge cutoff, what real-time should mean in hours, and when to use a contents or search API.
A language model answers from two places: what it learned in training and what you put in its context window. The first is frozen at a cutoff date. The second is the only way a model can know what a public web page says today. This guide explains when that matters, what "real-time" should mean in numbers, and which tool fits which job.
## What goes stale after the knowledge cutoff
Not every fact ages at the same speed. A practical way to sort the pages your app depends on:
| Page type | How fast it changes | What a stale answer looks like |
| --- | --- | --- |
| Status and incident pages | Minutes | "Everything is operational" during an outage |
| Pricing and plan pages | Days to weeks | An old price quoted to a customer |
| Docs, changelogs, release notes | Days | A removed parameter recommended in code |
| Policies and legal notices | Weeks to months | An outdated return or data policy |
| Reference and history pages | Rarely | Usually fine from training data |
The dangerous part is not that the model lacks the fact. It is that the model rarely says so. Asked about a page it has not seen in its current form, it reconstructs a plausible version from older training data and states it with the same confidence as everything else.
The fix is simple to describe: fetch the current page, put its clean content into the prompt, and tell the model where it came from and when it was read.
## What "real-time" should mean in numbers
"Real-time web data" is a phrase without a unit. In practice, every API that offers it is making one of three promises:
- **Live fetch:** the page is requested from the site now. It is as fresh as it gets and as slow as the site.
- **Cache within a max age:** a stored copy is returned if it is younger than a limit you set; otherwise the page is fetched live.
- **Cache only:** a stored copy of any age, or nothing.
Vendors express the limit in different units. Exa's contents endpoint uses `maxAgeHours`, Firecrawl's `maxAge` is in milliseconds, and others use seconds. The idea is the same: freshness is a number you choose per page type. The guide [live vs cached web data](/guides/live-vs-cached-web-data) has starting defaults for each type.
Ask any provider, including us, three questions: How old can the content be? How do I see when this result was read? What happens when the live fetch fails? Our own answers: every read is live, `statuses[].fetchedAt` gives the time, and a failure returns an error tag at 0 credits.
## Contents API vs search API
Two different jobs often hide behind "give the model web access":
1. **Finding pages.** You have a question, not a URL. A search API returns ranked URLs and snippets.
2. **Reading pages.** You have the URL. A contents API returns the page's content, cleaned for a model.
Many apps only need the second. A docs assistant knows its docs URLs. A pricing checker knows the pricing pages. A changelog summarizer knows where the changelog lives. Using a search engine for these adds cost and a ranking step that can return the wrong page.
If you need both, run them in order: search to pick URLs, then a contents call to read them. Do not expect one to replace the other.
## Public pages only
Live web data, as this site uses the term, means public pages: anything a visitor can read without logging in, paying or using a private API. That boundary is not a technicality. It is where the legal and ethical questions start, and the product's acceptable-use policy draws it in the same place.
Two habits help:
- Read the site's robots.txt (RFC 9309 describes the format) and its terms of service before you fetch from it at volume. robots.txt is a signal of the site owner's wishes, not an access control, and respecting it is part of being a good citizen on the web.
- Keep volumes proportionate. Fetching a page once an hour for a feature is different from hammering it.
This is general information, not legal advice.
## When a live fetch is the wrong tool
A live fetch is not always the answer:
- **The fact does not change.** Training data or a static reference is fine.
- **You need hundreds of related pages at once.** That is crawling or retrieval over an index, and the questions become chunking and refresh schedules. Our sister site has a guide on [web content ingestion for RAG](https://contextretrieval.net/guides/web-content-ingestion-for-rag) that covers what happens after the fetch.
- **The content only appears after JavaScript runs.** A static fetch returns little or nothing. Live Web Data API flags these pages (a `THIN_CONTENT` warning or a `CRAWL_NOT_RENDERED` error, which is not charged) and can read them in a browser when you set `render: true`.
- **You need structured fields extracted into a schema.** That is a separate step after reading the page.
## Build vs buy
A minimal fetcher is an HTTP GET and an HTML-to-Markdown library. A production one also needs:
- timeouts and retries that do not stall your users,
- main-content extraction, and a choice of format per task (see [Markdown, text or HTML for LLMs](/guides/markdown-text-or-html-for-llms)),
- a store of fetched copies keyed by URL and country, with a max age per source type,
- connections in each country whose version of a page you need (see [web pages by country](/guides/web-pages-by-country)),
- a clear error for each failure (not found, timed out, not rendered, unsupported content type),
- billing or quotas you can predict (see the [web data budget guide](/guides/web-data-budget-for-ai-apps)).
If that list is your product, build it. If it is plumbing under your product, buying it is usually cheaper than running it.
## Where Live Web Data API fits
Live Web Data API is the contents step: you send up to 10 public URLs per request and get each page back as Markdown, text or HTML. Every request reads the page live (there is no cache), `countryCode` sets where the page is read from, `statuses[].fetchedAt` records when each page was read, and `render` reads JavaScript-only pages in a browser. Failed or blocked requests cost 0 credits. It does not search, crawl or extract structured data. Sign-up is for businesses only. Only Business, Enterprise and Custom sign-ups are reviewed before checkout, usually within 1 business day. [Read the docs](/go/docs?from=post:why-llm-apps-need-live-web-data) or [request access](/go/signup?from=post:why-llm-apps-need-live-web-data).
## Checklist
- List the pages your app depends on and sort them by how fast they change.
- Decide per type: training data, a stored copy within a max age, or live.
- Confirm you know the URLs; add a search step only if you do not.
- Check that every target is public and that robots.txt and the terms allow your use.
- Pass the URL and fetch time into the prompt with the content.
- Decide what the app shows when a fetch fails.
---
# Live vs cached web data: where the cache belongs in LLM apps
Source: https://livewebdata.com/guides/live-vs-cached-web-data
> Live vs cached web data for LLM apps: how max age works across APIs, when to cache on your side, timeouts, and how to pass fetch times to the model.
Every copy of a web page has an age. A live fetch gives you age zero at the cost of waiting for the site. A cached copy is fast but might be hours or days old. Good LLM apps decide this per page type, keep the decision in configuration and send the age along with the content. This guide explains how vendor max-age controls work, when a cache on your own side pays off, and how to tell the model how old its source is.
## How max age works at providers that cache
Many contents APIs follow a pattern borrowed from HTTP caching (RFC 9111): you state the oldest copy you will accept.
- If the provider's cache holds a copy younger than your limit, you get it.
- If not, the page is fetched live and the cache is updated.
- A limit of zero means always live.
- Some APIs also offer cache only: never fetch, return what is stored or nothing.
Units differ between vendors. Exa uses hours (`maxAgeHours`), Firecrawl uses milliseconds (`maxAge`; `3600000` is one hour), and other APIs use seconds. Vendor parameters were checked against their public docs in October 2026; they change, so check again before you rely on them. When you compare providers or move between them, convert first: a max age of `24` means a day in one API and 24 milliseconds in another.
This site is published by Teneo Labs, which makes Live Web Data API, the third column in the table above. That API keeps no cache: every request reads the page live. It accepts `maxAgeHours: 0` so clients written for other providers keep working; any other value is rejected with a 400 instead of silently returning something older than you asked for.
## When a cache on your side pays off
With a live-only API, the cache moves into your app. That sounds like more work, but it puts the freshness decision where the knowledge is: you know which sources change hourly and which change once a quarter.
A small store is enough. Key it by URL and country, because a page read from Germany and the same page read from the US can differ; the guide on [web pages by country](/guides/web-pages-by-country) explains why. Store the content, the `fetchedAt` time from the response and the HTTP status. Before each read, compare the stored fetch time with the max age for that source type and only fetch live when the copy is too old.
| Page type | Starting max age | Why |
| --- | --- | --- |
| Status and incident pages | live, no stored copy | Minutes matter |
| Pricing, stock, availability | 1 hour | Changes are rare but costly to miss |
| News and announcements | 3 hours | Fresh, but not every minute |
| Docs, changelogs, release notes | 24 hours | Usually updated per release |
| Policies, terms, about pages | 168 hours | Change rarely and visibly |
These are starting points, not measurements. Two rules make the table work in production. First, put the max age in configuration per source, not in code. Second, log how often you serve a stored copy and how often you fetch live, so you can see whether the numbers fit. Every live fetch is a billed request, so a stored copy that is still young enough is also the cheapest read; the [web data budget guide](/guides/web-data-budget-for-ai-apps) covers that side.
## Latency: measure, do not assume
A read from your own store returns as fast as your database. A live fetch takes as long as the target site plus extraction. The difference depends on the site, the country and the time of day, so published averages tell you little about your pages.
Measure it yourself: fetch a sample of your real sources live and look at `searchTime` in each response. That gives you numbers to design timeouts and user-facing waits around.
## Timeouts and fallback to a stored copy
A live fetch can hang. `livecrawlTimeout` caps the wait in milliseconds; the default is 10,000 and the maximum is 90,000. When the limit is reached, the result is the error `CRAWL_LIVECRAWL_TIMEOUT`, and failed fetches cost 0 credits.
If you keep stored copies, you have a middle path: when the live fetch fails, serve the last stored copy and mark it as older. That is often the right answer for a user waiting on a response, but it must be visible:
- Show the fetch time next to the answer ("page read 2 days ago").
- For high-stakes pages such as prices, treat a failed live fetch as an error and say the current page could not be read.
## Record the fetch time of every page
Each URL in a request gets an entry in `statuses[]`. On success, `source` is `livecrawled` and `fetchedAt` holds the time the page was read during this request. Store both next to the content, together with the country. It is the difference between "we think this is current" and "we know when this was read".
## Tell the model how old the page is
The model cannot see HTTP headers or your store. If freshness matters to the answer, put it in the prompt:
```text
...page markdown...
Answer from the source above. If the question depends on information newer than the fetch time, say so.
```
Use the `fetchedAt` value from the response, or the time you stored the copy. Do not rely on the page's `Last-Modified` header: dynamically generated pages often report the current time, and some static pages never update it. RFC 9111 treats such dates as heuristics for exactly this reason.
## Where Live Web Data API fits
The API is the live half of this design: each request reads the page now, returns `statuses[].fetchedAt`, and caps the wait with `livecrawlTimeout`. Failed fetches, including timeouts, cost 0 credits. You can try live reads on your own URLs in the [playground](/go/playground?from=post:live-vs-cached-web-data) once your plan is active, or read the response fields in the [API reference](/go/api-reference?from=post:live-vs-cached-web-data).
## Checklist
- Every source type has a max age, stored in configuration.
- Stored copies are keyed by URL and country and keep their fetch time.
- Units are converted when comparing or switching providers.
- `livecrawlTimeout` matches what your users can wait for.
- A stored copy served after a failed live fetch is shown with its age, or treated as an error, by page type.
- The fetch time and country travel with the content into the prompt.
---
# Country-specific web content for LLMs: same URL, other page
Source: https://livewebdata.com/guides/web-pages-by-country
> Why the same URL returns different content by country, what that does to LLM answers, and how to pick a default country and label localized pages.
Open a public pricing page from Berlin and from New York and you may get two different pages at the same URL: different language, currency, consent banner and legal notice. An LLM that reads one of them and answers a user in the other country gives a confidently wrong answer. This guide explains why it happens, which country to read from, and how to keep the model honest about it.
## Why the same URL differs by country
Sites adapt pages to the reader in a few common ways:
- **By the connection's location.** The server or its CDN looks at where the request comes from and serves a regional variant or redirects to one.
- **By language headers.** The `Accept-Language` header states preferred languages. Sites that honor it may switch language without changing the URL.
- **By regulation.** Consent banners, cookie notices and legal disclosures differ by jurisdiction, so a page read from the EU often carries text that the US version does not.
The HTTP `Vary` header sometimes signals that a response depends on a request header, but many sites vary by location without saying so. The only reliable way to know what readers in a country see is to read the page from there.
## What it does to an LLM answer
Country differences show up in answers in predictable ways:
| What differs | Typical wrong answer |
| --- | --- |
| Currency | A US user is quoted a euro price |
| Plans or products | A plan is recommended that is not sold in the user's market |
| Legal terms | Return or cancellation rules from another jurisdiction |
| Language | A summary that mixes languages or mistranslates terms |
| Consent text | Banner text leaks into the summary as if it were content |
None of these are model failures. The model answered from the page it was given.
## Choose a default country on purpose
A simple decision order works for most apps:
1. **The user's country**, if your app knows it and the question is about something they would buy, use or comply with.
2. **The country the question names**, if it names one ("What does the German page say about returns?").
3. **Your product's main market** as the fallback default.
Write the default down in configuration. Live Web Data API reads pages from the US unless you set `countryCode`; the code is an ISO 3166-1 alpha-2 value such as `de`, `jp` or `br`. Every request reads the page live from the country you set, so there is no shared copy to mix up. If you store copies yourself, key them by URL and country; the guide on [live vs cached web data](/guides/live-vs-cached-web-data) covers how old those copies may be.
## Consent banners in fetched pages
Pages read from the EU often include a cookie consent layer. With `verbosity: "compact"` (the default) the API returns the main content only; `"full"` keeps the whole body, including navigation and footer. Consent text can still get through: some sites build the banner into the page body, and then fragments of its text can appear in the main content. The guide on [Markdown, text or HTML for LLMs](/guides/markdown-text-or-html-for-llms) covers when the full page is worth its tokens. Two mitigations:
- Strip known banner phrases in a post-processing step for sources where you have seen them.
- Ask the model to ignore consent and cookie text when summarizing.
Use `verbosity: "full"` only when you need the whole page, for example to check what a banner or footer says.
## Pass the country to the model
Put the country next to the content, just like the URL and fetch time:
```text
...page markdown...
Answer for readers in the country shown on the source. State prices in the currency used on the page.
```
If a question really is a comparison ("How do the US and German pages differ?"), fetch both and label each source clearly.
## Check your own localized pages
The same technique is a cheap QA tool. Fetch your own public pages from each country you serve and compare the Markdown:
- Is the right language served, or does one market get the fallback?
- Do prices and currencies match what you intend for each country?
- Does any market see an outdated legal notice?
Because Markdown drops markup, the differences are easy to diff.
## hreflang is not what a fetcher receives
`hreflang` annotations tell search engines which localized URLs exist for a page. They do not change what a server sends to a given request. A fetcher reading `https://example.com/pricing` from Germany receives whatever the site serves to that request. If you want a specific localized version, request its own URL, and if the site varies the page by location, read it from the matching country.
## Public pages, by the rules
A country setting shows you the public page as it is served in that country. It is not a way to reach content that a site restricts by login, paywall or other access control; that is excluded by the product's acceptable-use policy and is not a topic on this site. Check robots.txt and the site's terms before fetching at volume. This is general information, not legal advice.
## Where Live Web Data API fits
Set `countryCode` per request and the page is fetched through a connection in that country; the default is `us`. The result comes back as Markdown, text or HTML with the same per-URL `statuses[]` as any other request. See how `countryCode` works in the [API reference](/go/api-reference?from=post:web-pages-by-country), or [request access](/go/signup?from=post:web-pages-by-country) as a business.
## Checklist
- The app has a written default country and a rule for when to override it.
- Every fetched page is stored with its country.
- The country goes into the prompt with the content.
- Consent text is filtered for sources where it leaks through.
- Comparisons across countries label each source.
- Your own localized pages are checked from each market you serve.
---
# Web data budget for AI apps: fetch cost plus token cost
Source: https://livewebdata.com/guides/web-data-budget-for-ai-apps
> Web data budget for AI apps: a worksheet for both bills, the fetch and the input tokens, plus truncation, failed requests, batching and rate limits.
Every page that enters a model's context is paid for twice: once to fetch it and once as input tokens. Most cost estimates count only the first. This guide gives you a worksheet that counts both, then covers the levers that move each one.
## The two bills
**The fetch bill** is what your web data provider charges per page. With Live Web Data API, every successful read is a live fetch and is billed; how many credits a page takes depends on how it had to be fetched (a standard page, a page behind bot protection, or a browser-rendered page), and a failed one costs 0. Current plans and credit amounts are on the [pricing page](/go/pricing?from=post:web-data-budget-for-ai-apps).
**The token bill** is what your model provider charges for the page's content as input. It scales with page length and format. A page passed as raw HTML carries scripts, attributes and navigation; the same page as main-content Markdown carries mostly the words and structure that matter.
## The worksheet
Fill in your own numbers. Label every assumption so you can replace it with a measurement later.
| Line | Formula | Notes |
| --- | --- | --- |
| A. Questions per month | from your product forecast | |
| B. Pages per question | average | 1 for a single-source answer |
| C. Page reads | A × B | |
| D. Success rate | from logs, or an assumption | failures are 0 credits here |
| E. Billed fetches | C × D, minus reads served from your own store | every live success counts |
| F. Fetch cost | E × credits per page × price per credit from your plan | rendered and protected pages take more credits |
| G. Tokens per page | measured on your sources | after cleaning and truncation |
| H. Token cost | C × G × model input price per token | every read enters the context |
| I. Total | F + H, plus headroom for peaks | |
Two notes. Live Web Data API has no cache, so the only way to bill fewer fetches is to read fewer pages live: store copies on your side and reuse them within a max age. If your provider caches and bills cache hits differently, add a line for it from its billing docs. And if a fetch fails, you pay neither bill for that page, but your app still has to answer.
## Measure tokens per page yourself
Do not guess line G. Take ten representative URLs, fetch each as Markdown with your usual settings, and count tokens with your model's tokenizer: `tiktoken` for OpenAI models, the token counting endpoint for Anthropic models. Then fetch the same pages as HTML and count again. The difference is what your format choice is worth, and the guide on [Markdown, text or HTML for LLMs](/guides/markdown-text-or-html-for-llms) explains the trade-offs.
## Lever 1: truncate long pages
Long pages are where the token bill grows (the [Markdown, text or HTML guide](/guides/markdown-text-or-html-for-llms) covers the format side). `text.maxCharacters` truncates Markdown and text to a limit you set; the default is 100,000 characters, and `results[].textTruncated` says when a page was cut. The post on the [token cost of web pages](/blog/token-cost-of-web-pages) shows how to turn a token ceiling into that limit. Choose the limit by page type:
- Product, pricing and status pages: the answer is usually near the top, so a few thousand characters is often enough.
- Docs and changelogs: keep more, or fetch the specific section URL instead of the whole page.
- Long articles: truncate for classification, keep more for summaries.
Test the limit on your own pages: check that the answers you need survive the cut.
## Lever 2: store copies and set a max age
A max age on copies you store stops the same page from being fetched live again and again within minutes. That makes answers faster, keeps your load on the target site proportionate and, because Live Web Data API bills every live read and has no cache of its own, cuts line E directly. Keep the `fetchedAt` time with each stored copy so you know its age. The guide on [live vs cached web data](/guides/live-vs-cached-web-data) has starting defaults per page type.
## Lever 3: send fewer pages
The cheapest page is the one you do not send. Only include pages the question needs, and drop duplicates that redirect to the same final URL (the response includes `results[].url`, the final URL after redirects).
## Failed requests
Failures are normal on the open web: pages move, sites time out, some content needs JavaScript. How they are billed varies by provider, so read the billing docs. In Live Web Data API, timeouts, not-found pages, restricted targets and pages without usable content all cost 0 credits, and each comes back with an error tag such as `CRAWL_NOT_FOUND` or `CRAWL_LIVECRAWL_TIMEOUT`. That keeps a retry cheap, but retries still add latency, so cap them.
## Batching and rate limits
Up to 10 URLs fit in one request. Batching saves round trips when you know several URLs up front; each page is still billed on its own. A URL that fails to fetch does not fail the others; invalid parameters or too few credits fail the whole request.
Rate limits are set per plan and are listed on the pricing page. Each URL in a batch counts toward the per-second limit, so a 10-URL request uses ten units, not one. Responses carry `X-RateLimit-Limit` and `X-RateLimit-Remaining` headers, and a `429` comes with `Retry-After`. If your peak is bursty, put a queue in front of the fetch so a spike waits instead of failing.
## Where Live Web Data API fits
The product's billing is simple to model: every successful read is live and billed by how the page had to be fetched, and failures cost 0. Sign-up is for businesses only. Only Business, Enterprise and Custom sign-ups are reviewed before checkout, usually within 1 business day. [See current plans](/go/pricing?from=post:web-data-budget-for-ai-apps) and plug them into line F.
## Checklist
- Both bills are in the budget: fetch and tokens.
- Tokens per page are measured on your sources, in your format.
- Each page type has a character limit and a max age.
- Failure billing is confirmed in your provider's docs.
- Rate limits and peaks are planned with a queue and retries capped.
- Assumptions are labelled and replaced with measurements after launch.
---
# Markdown vs HTML for LLMs: which page format to pass
Source: https://livewebdata.com/guides/markdown-text-or-html-for-llms
> Markdown vs HTML for LLM context, plus plain text: what each format keeps, how much of the page to pass, tables, metadata and which URL to cite.
A model reads exactly what you give it. Pass a page as raw HTML and it reads class names, scripts and menus along with the article. Pass it as clean Markdown and it reads the headings, paragraphs, lists and links. This guide covers what each format keeps, when the full page matters, and which metadata to send with the content.
## The three formats
**Markdown** keeps the structure a reader uses: headings, lists, emphasis, links, code blocks and simple tables. It drops tags, attributes and scripts. For reading, summarizing and citing, it is the best default.
**Plain text** keeps only the words, with whitespace normalized. It is the cheapest in tokens (the [web data budget guide](/guides/web-data-budget-for-ai-apps) shows how to put that token bill into numbers) and the right choice when structure does not matter: classification, language detection, keyword checks.
**HTML** keeps the page as served. It costs the most tokens and carries the most noise, but it is the only format that preserves attributes, forms and complex tables. Use it when the structure is the data.
There is an honest counterargument: some practitioners report that cleaned HTML works well because its closing tags make structure explicit. That is a reasonable choice for structure-heavy pages. The bigger gain, in either format, comes from removing what is not content.
## Main content vs full page
The second decision is how much of the page to pass. It matters as much as the format.
- **Main content** (Live Web Data API's `verbosity: "compact"`, the default) returns the main content only. Use it for almost everything.
- **Full page** (`verbosity: "full"`) keeps the whole body, including navigation and footer. Use it when the question is about those parts: a legal notice in the footer, a site's menu structure, or a banner's wording.
Navigation text in the context is not just a cost problem. Menus full of product names and "related articles" lists give the model plausible-looking text to answer from that is not the page's topic.
## Tables
Simple tables convert cleanly to GitHub Flavored Markdown: one header row, one value per cell. Conversion struggles with:
- merged cells (`rowspan`, `colspan`),
- nested tables,
- tables used for page layout rather than data,
- cells that contain lists or several paragraphs.
If a page's value is in a complex table, for example a detailed feature matrix, request HTML for that page and let the model read the table markup, or parse the table yourself before prompting.
## Metadata worth keeping
The content is half the story. Send the model the context that lets it judge and cite the source:
| Field | Why it matters |
| --- | --- |
| Final URL | What to cite; the address after redirects |
| Title | Lets the model refer to the source by name |
| Published date | Distinguishes an old article from a current one, when the page exposes it |
| Author | Useful for attribution, when the page exposes it |
| Fetch time | How current the content is; record it yourself |
| Country | Which regional version was read |
Live Web Data API returns `url`, `title`, `author`, `publishedDate`, `image` and `favicon` per result, with `null` when a page does not expose a value. Do not invent missing metadata; pass `null` or leave the field out.
## Cite the final URL
Requests are often redirected: to HTTPS, to a trailing slash, to a localized path or to a page's new home. The address you requested is not always the address a reader should open. Use the final URL (`results[].url`) when you show sources, and keep the requested URL (`results[].id`) for your own logs and cache keys.
## JavaScript-only pages
Some pages send almost no content in their HTML and build it in the browser. A static fetch cannot turn those into useful Markdown. Live Web Data API flags them with a `THIN_CONTENT` warning or a `CRAWL_NOT_RENDERED` error (0 credits); set `render: true`, or `renderFallback: true` to render only when needed, and the page is read in a browser before it is converted. Our sister site has a guide on [whether you need a headless browser](https://siterender.cc/guides/do-you-need-a-headless-browser) for a given page.
## Where Live Web Data API fits
Each successful result includes the page as Markdown and plain text, and as HTML when you set `html: true`, so you can choose per task without fetching twice. `text.verbosity` switches between main content and full page, and `text.maxCharacters` caps the length of Markdown and text. Compare the formats on your own pages in the [playground](/go/playground?from=post:markdown-text-or-html-for-llms), or see the [response fields in the API reference](/go/api-reference?from=post:markdown-text-or-html-for-llms).
## Checklist
- Markdown is the default; HTML is an explicit per-page exception.
- Plain text is used only where structure does not matter.
- Main content is the default; full page only when the question is about navigation or footers.
- Pages with complex tables are flagged and handled as HTML.
- Final URL, title, fetch time and country travel with the content.
- Missing metadata stays missing.
---
# Tell the model how old the page is: fetch timestamps in context
Source: https://livewebdata.com/blog/fetch-timestamps-in-llm-context
> A model cannot see when a page was read. Put the fetch time, URL and country next to the content, in one format, and tell the model how to use them.
A model reads a fetched page as if it were timeless. Nothing in the Markdown says "read at 09:14 UTC today", so the model cannot tell a page fetched a minute ago from one fetched last month. If the answer depends on a price, a status or a changelog, that difference decides whether the answer is right. The fix is cheap: put the time next to the content, in one format, and say what it means.
## Which time to record
There are three candidate times, and only one of them is yours:
- **Fetch time.** When your app (or your provider) read the page. This is the only time you can vouch for.
- **The page's published or updated date.** What the page says about itself. Useful, but it is the site's claim, and many pages do not expose one.
- **The HTTP `Date` header.** MDN describes it as the time the response was generated. It is close to the fetch time, but it comes from the target server's clock and is not shown to you by every API.
Record the fetch time for every page, and the published date only when the page exposes it.
## One format: RFC 3339 in UTC
RFC 3339 defines the timestamp profile most APIs already use, for example `2026-10-27T09:14:05Z`. In JavaScript, `Date.prototype.toISOString()` returns this form, always in UTC with the `Z` suffix. Use it everywhere: in your logs, your cache keys and the prompt. One format means the model never has to guess whether `10/11` is October or November.
## A header block per page
Put a short block above each page's content:
```text
[source 2]
url: https://example.com/status
fetched_at: 2026-10-27T09:14:05Z
country: de
page_published: not stated
---
(page Markdown)
```
Then give the model today's date once, at the top of the prompt, and one instruction: "Each source shows when it was fetched. If a fact may have changed since then, say when the source was read." That turns hidden staleness into a sentence the user can see.
## Common mistakes
- **Local time without an offset.** `09:14` means nothing without a zone. Always write `Z` or an offset.
- **Using the page's date as the fetch time.** A page "updated in 2024" that you read today is a page fetched today with old content. Keep both, labelled.
- **Dropping the time when you truncate.** If you cut a page to fit a budget, keep the header block.
The [live vs cached web data guide](/guides/live-vs-cached-web-data) covers the freshness decision this timestamp belongs to.
## How this maps to Live Web Data API
Each URL in a response has a `statuses[].fetchedAt` in ISO 8601, set when the page was read, and `null` when the fetch failed. `results[].url` is the final URL after redirects, `results[].publishedDate` is filled when the page exposes one, and `countryCode` is the country you asked for. Those four fields fill the header block above without extra code. The [API reference](/go/api-reference?from=post:fetch-timestamps-in-llm-context) lists every field.
---
# Final URL or canonical URL: which address an LLM should cite
Source: https://livewebdata.com/blog/final-url-vs-canonical-url
> A fetched page has up to three addresses. Which one to pass to the model, which one to key stored copies by, and what redirects reveal about country and age.
A fetched page arrives with an address, and that address is metadata the model uses. It decides what the model cites, it often tells you which country's version of the page you read, and it is the cheapest way to spot duplicates before they cost input tokens. For a fetched web page there are usually three candidate addresses, and they are often different strings: the URL you requested, the URL the request ended at, and the URL the site says is the real one.
## Requested URL: the key for your logs and stored copies
The requested URL is what your code asked for. It is the right key for your own logs, retries and stored copies, because it is the input you control. Key stored copies by requested URL and country, since the same URL can return a different page from another country. It is not always what a reader should open: it may be an old address, a tracking link or a short link.
## Final URL: the version you actually read
Servers move pages with redirects. MDN groups them by intent: permanent ones (`301`, `308`) say the resource has a new home; temporary ones (`302`, `303`, `307`) say "look over there for now". A fetch follows the chain and reads the page at the last hop.
That last address is the honest citation. It is where the content in your context came from, and it is where a reader who clicks the citation will land. It also carries information the requested URL hides:
- **Country and language.** A redirect to `/de/` or `fr.example.com` shows which localized version was served. Pass it with the content, so the model does not answer a US user from a German page. The guide on [web pages by country](/guides/web-pages-by-country) covers why the served page changes.
- **Age.** With temporary redirects, the next fetch may end somewhere else. Store the fetch time next to the final URL, so the citation says when it was true.
## Canonical URL: what the site prefers
A page can declare its preferred address with ``. RFC 6596 defines this link relation for pages whose content is duplicated or very similar at several URLs. Google's guidance treats it as a strong signal for choosing among duplicates, not as an order it must follow.
For an LLM app that makes the canonical URL useful in one place: grouping. If three fetched pages declare the same canonical, they are probably one source. Sending all three to the model costs input tokens three times, and the model may count them as three independent confirmations. Drop the copies before you build the prompt.
## A rule you can apply
- **Pass to the model** the final URL, the fetch time and the country, next to the content.
- **Log and key stored copies** by the requested URL and the country.
- **De-duplicate** by the canonical URL when the page declares one, and fall back to the final URL when it does not.
The [Markdown, text or HTML guide](/guides/markdown-text-or-html-for-llms) covers the other metadata worth passing to the model with the content.
## How this maps to Live Web Data API
Each result carries `results[].id`, the URL you requested, and `results[].url`, the final URL after redirects. `countryCode` sets the country the page is read from, and `statuses[].fetchedAt` records when it was read. The canonical link is not a separate field; set `html: true` and `results[].html` carries the page HTML as served, so you can read the `rel="canonical"` link from it when you need to group duplicates. You can compare the fields on your own pages in the [playground](/go/playground?from=post:final-url-vs-canonical-url) once your plan is active.
---
# Last-Modified and ETag: what page headers say about freshness
Source: https://livewebdata.com/blog/last-modified-and-etag-freshness-hints
> Last-Modified and ETag tell a cache whether a copy changed, not when the content was written. How reliable each one is, and what to store instead.
HTTP gives every response two optional freshness hints: the `Last-Modified` date and the `ETag` version tag. It is tempting to read them as "this page was updated on…" and pass that to the model. Both headers were designed for a narrower job, and knowing that job tells you how far to trust them.
## What the headers are for
RFC 9110 calls both headers *validators*. Their job is to let a cache ask the origin server a cheap question: "I have a copy; has it changed?" The cache sends `If-Modified-Since` with the stored date or `If-None-Match` with the stored ETag. If nothing changed, the server answers `304 Not Modified` without a body, and the cache keeps using its copy.
That is a statement about the *response*, as the server sees it. It is not a statement about when the article, price or policy on the page was written.
## Where Last-Modified goes wrong
MDN describes `Last-Modified` as the date the origin server believes the resource was last modified, with one-second resolution. "Believes" is doing a lot of work:
- **Generated pages** often send the time of the request, so every fetch looks brand new.
- **Static site builds** can stamp every file with the build time, so a page untouched for a year looks as fresh as yesterday's post.
- **Some servers omit it**, and some CDNs rewrite it.
For an LLM app, the result is the same in every case: the header cannot tell the model whether the facts on the page are current.
## Where ETag goes wrong
An `ETag` is an opaque identifier for one version of a resource. MDN and RFC 9110 both distinguish strong tags from weak ones (prefixed `W/`), which only promise that two responses are equivalent for caching. The value has no defined format, so you cannot read a date or a version number out of it. It changes when anything in the response changes, including parts the model does not care about, such as a rotating banner.
## What to record instead
Treat validators as hints for caching, and record the facts you can vouch for:
- **Fetch time**, taken by your own code when the response arrives.
- **Source status**: whether the copy was read live or came from a cache (yours or a provider's), and how old it was allowed to be.
- **Published date from the page**, when the page exposes one, labelled as the page's own claim.
- **Final URL and country**, so the age belongs to a specific version of the page.
Then tell the model the fetch time in the prompt, as the [live vs cached web data guide](/guides/live-vs-cached-web-data) shows.
## How this maps to Live Web Data API
The API does not pass the target site's response headers through. It returns the content, a `publishedDate` when the page exposes one, and per URL a `statuses[].fetchedAt` with the time the page was read. There is no cache, so every result was read live during your request. Store `fetchedAt` next to the content. The [API reference](/go/api-reference?from=post:last-modified-and-etag-freshness-hints) lists every response field.
---
# Token cost of web pages: how many tokens one page really is
Source: https://livewebdata.com/blog/token-cost-of-web-pages
> In one 44,684-page sample the median page was 3,201 tokens and the 99th percentile 141,410. How to budget the token bill when page sizes skew that far.
Every page that enters a context window is paid for twice: once to fetch it and once as input tokens. The [web data budget guide](/guides/web-data-budget-for-ai-apps) calls the second one the token bill and leaves one line for you to fill in: tokens per page. That line is the hardest to guess, because web pages do not have a typical size. They have a median, a mean far above it, and a tail more than 40 times longer than the median page.
## What one large sample measured
In December 2025, DEJAN AI published token counts for 44,684 web pages: articles, papers, product pages, documentation and PDFs, counted with Gemini's tokenizer.
| Point in the distribution | Tokens per page |
| --- | --- |
| 25th percentile | 1,396 |
| Median | 3,201 |
| 75th percentile | 8,207 |
| Mean | 10,403 |
| 90th percentile | 21,839 |
| 95th percentile | 35,852 |
| 99th percentile | 141,410 |
Two limits before you reuse these numbers. The write-up does not say how the pages were chosen, and it does not say whether a count covers only a page's main content or everything on it. Treat the table as the shape of the problem, not as a figure for your own pages.
The shape is the useful part. The mean is more than three times the median. A budget built on the mean overstates a typical request; one built on the median misses what the long pages cost.
## Format moves the number more than the page does
The same page costs very different amounts depending on what you send. Cloudflare's February 2026 launch post for its Markdown for Agents feature uses itself as the example: 16,180 tokens as HTML and 3,150 tokens as Markdown. That is one page, measured by the vendor that sells the conversion, so read it as an illustration. It shows the mechanism, though: Markdown keeps headings, lists and links and drops tags, attributes and scripts. The [Markdown, text or HTML guide](/guides/markdown-text-or-html-for-llms) covers what each format keeps.
A third variable is the tokenizer. The sample above was counted with Gemini's; other models split the same text into a different number of tokens, so a budget measured on one model is a starting point for the next one, not a result.
## Budget the median, cap the tail
A rule that holds up when sizes skew this far:
1. **Estimate with the median** of your own sample, times page reads per month. That is your typical bill.
2. **Cap the tail** with a character limit per page type, so one very long page cannot cost as much as 40 typical ones.
3. **Check against the mean** after the cap. If the capped mean is still far above the median, the cap is too loose for your sources.
A worked example with one labelled assumption: an app reads 10,000 pages a month. At the sample's median, that is 32,010,000 input tokens (10,000 × 3,201). At the sample's mean, it is 104,030,000 (10,000 × 10,403). Same pages, a threefold difference in the forecast, depending only on which number you picked. Multiply both by your model's input price per million tokens, and plan for the higher figure until a cap and your own measurements bring it down.
## Turn a token ceiling into a character cap
The budget guide shows how to measure tokens per page on ten of your own URLs. Keep two more numbers from that same sample:
1. **The longest page.** If it is far above the median, your sources have a tail like the one above, and a cap will matter.
2. **Characters per token.** Divide each page's character count by its token count and take the median. It differs by language and by how much code or markup a page carries, so measure it rather than borrowing a rule of thumb.
Then pick a token ceiling per page type and multiply it by your characters per token. That product is the character cap to set. In a distribution like the sample's, a ceiling near the 75th percentile (8,207 tokens) leaves about three in four pages untouched and cuts only the long ones. Check after the cut that the answers you need are still on the page.
## What this means for your app
- Put tokens per page in the budget as a measured median, not a guess.
- Set a character cap per page type: tight for pricing and status pages, looser for docs.
- Recount when you change models or formats.
- Record when a page was cut, so a truncated copy never passes as a complete one.
## How this maps to Live Web Data API
Each result carries the page as `markdown` and `text`, and as `html` when you set `html: true`, so one fetch gives you every format to count. `text.verbosity: "compact"`, the default, returns the main content only. `text.maxCharacters` cuts Markdown and text at the limit you set, and `results[].textTruncated` is `true` when it did. Failed fetches cost 0 credits. The [API reference](/go/api-reference?from=post:token-cost-of-web-pages) lists every option and its limits.
---
# Live Web Data API — full documentation
# Overview
Live Web Data API turns any public URL into clean Markdown, text or HTML — fetched live through our fetch network.
## What it does
You send up to 10 URLs (and optionally a country). We fetch each page through a residential or mobile connection, extract the main content and return it in the format your agent or pipeline needs. Successful pages cost 1 credit; failures cost nothing.
| | Value |
| --- | --- |
| Base URL | `https://livewebdata.com/api` |
| Endpoints | `POST /v1/contents`, `GET /v1/account` |
| Formats | markdown and text by default; html with `html: true` |
| OpenAPI | [`/openapi.json`](https://livewebdata.com/openapi.json) · [`/openapi.yaml`](https://livewebdata.com/openapi.yaml) |
| Fetch strategy (v1) | static fetch (`fast`), a retry through another connection (`retry`), or a real browser (`browser`) with `render` or `renderFallback` |
## Authentication
Create a key in the dashboard under [API keys](https://livewebdata.com/dashboard/keys) and send it as a bearer token. Keys start with `1337_live_` (test keys with `1337_test_`) and are shown only once. Keep them in an environment variable such as `LIVE_WEB_DATA_API_KEY` — never in client-side code or repositories.
```bash title="Headers"
Authorization: Bearer 1337_live_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Content-Type: application/json
```
## Quickstart
Fetch a page as Markdown:
```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
-H "Authorization: Bearer $LIVE_WEB_DATA_API_KEY" \
-H "Content-Type: application/json" \
-d '{"urls": ["https://example.com/article"]}'
```
The response contains one entry in `results` per successful page and one entry in `statuses` per requested URL. The page HTML is included only when you send `"html": true`:
```json title="Response (abridged)"
{
"requestId": "req_01J…",
"results": [{
"id": "https://example.com/article",
"url": "https://example.com/article",
"title": "Article",
"markdown": "# Article\n\n…",
"text": "Article\n\n…"
}],
"statuses": [{
"id": "https://example.com/article",
"status": "success",
"source": "livecrawled",
"fetchedAt": "2026-10-23T09:15:02.118Z",
"httpStatusCode": 200,
"strategy": "fast",
"credits": 1,
"error": null
}],
"searchTime": 1240
}
```
When `text` or `markdown` was cut at `text.maxCharacters`, `results[].textTruncated` is `true`. `results[].warnings` (absent when empty) flags pages that could only be partly retrieved (`PAGE_TRUNCATED`), probably need `render: true` (`THIN_CONTENT`), were converted as plain text (`EXTRACTION_FALLBACK`) or lost their `html` (and maybe part of their text) to the response's 24 MiB size limit (`RESPONSE_SIZE_LIMIT`).
See the [API reference](https://livewebdata.com/docs/api-reference) for every parameter, or try it without code in the dashboard [playground](https://livewebdata.com/dashboard/playground).
## SDKs
Thin, typed clients for JavaScript/TypeScript and Python with built-in retries and typed errors. See [SDKs](https://livewebdata.com/docs/sdks).
```javascript title="JS SDK"
import { Teneo } from "@teneo/contents";
const client = new Teneo({
apiKey: process.env.LIVE_WEB_DATA_API_KEY,
baseUrl: "https://livewebdata.com/api",
});
const res = await client.contents({
urls: ["https://example.com/article"]
});
for (const s of res.statuses) console.log(s.id, s.status, s.error?.tag ?? "");
console.log(res.results[0]?.markdown);
```
```python title="Python SDK"
import os
from teneo import Teneo
client = Teneo(
api_key=os.environ["LIVE_WEB_DATA_API_KEY"],
base_url="https://livewebdata.com/api",
)
res = client.contents(
urls=["https://example.com/article"],
)
for s in res["statuses"]:
print(s["id"], s["status"], (s["error"] or {}).get("tag", ""))
print(res["results"][0]["markdown"] if res["results"] else "")
```
## Live fetching
Every request is fetched live; there is no cache.
## Geotargeting
Set `countryCode` to an ISO-3166-1 alpha-2 code (for example `de`, `jp`, `br`) to fetch the page as seen from that country. The default is `us`.
```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"urls": [
"https://example.com/pricing"
],
"countryCode": "de"
}'
```
```python title="Python"
import requests
response = requests.post(
"https://livewebdata.com/api/v1/contents",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json={
"urls": ["https://example.com/pricing"],
"countryCode": "de"
},
timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```
```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
method: "POST",
headers: {
Authorization: "Bearer YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({
"urls": [
"https://example.com/pricing"
],
"countryCode": "de"
}),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```
## Multiple URLs
Send up to 10 URLs in one request. They are processed in parallel; `statuses` has one entry per URL (matched by `id`) and credits are charged per successful URL. A restricted URL gets a per-URL `RESTRICTED_TARGET` status instead of failing the whole request. More than 10 URLs return `400 TOO_MANY_URLS`. Rate limits count URLs, not HTTP requests.
```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"urls": [
"https://example.com/a",
"https://example.com/b",
"https://example.com/blocked"
]
}'
```
```python title="Python"
import requests
response = requests.post(
"https://livewebdata.com/api/v1/contents",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json={
"urls": ["https://example.com/a", "https://example.com/b", "https://example.com/blocked"]
},
timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```
```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
method: "POST",
headers: {
Authorization: "Bearer YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({
"urls": [
"https://example.com/a",
"https://example.com/b",
"https://example.com/blocked"
]
}),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```
## Restricted targets
Some domain categories are blocked for legal and regulatory reasons. Requests are rejected with `451` and cost no credits. Repeated attempts lead to automatic key suspension. Report abuse to abuse@livewebdata.com.
## MCP for agents
Agents can use the API as a tool over MCP with your existing key — two tools, `get_page_contents` and `get_account_status`. Setup for Claude Code, Cursor, VS Code and Claude Desktop: [MCP for agents](https://livewebdata.com/docs/mcp).
## For AI coding agents
Every docs page is also available as plain Markdown by appending `.md` to its URL (e.g. [`/docs/api-reference.md`](https://livewebdata.com/docs/api-reference.md)). An index for LLMs lives at [`/llms.txt`](https://livewebdata.com/llms.txt) and the full docs in one file at [`/llms-full.txt`](https://livewebdata.com/llms-full.txt).
---
# API reference
Two endpoints. HTTP 200 is returned even when individual pages failed — check statuses[] for the per-URL outcome.
## Endpoint
```bash title="Request"
POST https://livewebdata.com/api/v1/contents
Authorization: Bearer
Content-Type: application/json
```
Machine-readable specs:
- OpenAPI 3.1 (JSON): https://livewebdata.com/openapi.json
- OpenAPI 3.1 (YAML): https://livewebdata.com/openapi.yaml
- Postman collection: https://livewebdata.com/live-web-data-api.postman_collection.json
- Interactive reference: https://livewebdata.com/docs/api-reference/interactive
## Request parameters
| Parameter | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `urls` | `string[]` | yes | — | The pages to fetch, 1–10 http or https URLs of at most 8,192 characters each. Processed in parallel; one entry per URL in results/statuses. Duplicates (same URL, ignoring #fragment) are fetched once; repeated positions cost 0 credits. |
| `countryCode` | `string` | no | "us" | ISO-3166-1 alpha-2 code. The page is fetched through a connection in this country. |
| `text` | `boolean \| object` | no | true | Content options. Pass an object to set maxCharacters and verbosity. |
| `text.maxCharacters` | `integer` | no | 100000 | Truncates text and markdown to this many characters (max 1,000,000); `results[].textTruncated` tells you when it did. The MCP tool defaults to 8,000 (`max_characters`). |
| `text.verbosity` | `"compact" \| "full"` | no | "compact" | compact = main content only. full = whole body incl. navigation and footer. |
| `html` | `boolean` | no | false | Include the page HTML as served in results[].html. A response carries at most 24 MiB of JSON: past that, the remaining results come without html (warning RESPONSE_SIZE_LIMIT). |
| `livecrawlTimeout` | `integer (ms)` | no | 30000 (60000 with render) | Maximum time for a live fetch, 1–90000 ms. A lower value is raised to 30000, the time the browser fallback needs, unless you send renderFallback: false (then any value, default 10000). With render or renderFallback: true at least 30000. |
| `render` | `boolean` | no | false | Render every URL in a real browser, as a normal visitor sees it: 3 credits, 5 when the page needed a retry through another connection. Cannot be combined with renderFallback. |
| `renderFallback` | `boolean` | no | on | Fetch normally first and render in a real browser only when the first attempt did not return the page, shows no or very little visible text, or was blocked. On by default: omit it. Send false for a plain fetch only (never a browser). A normal success costs its normal 1 or 2 credits, a rendered one only the render credits; statuses[].fallbackUsed tells you when it rendered. Cannot be combined with render: true. |
```json title="Example body"
{
"urls": ["https://example.com/article"],
"countryCode": "de",
"text": { "maxCharacters": 50000, "verbosity": "compact" },
"livecrawlTimeout": 45000
}
```
Unknown fields are rejected with `400 INVALID_REQUEST`; the message names the field. So is a URL longer than 8,192 characters (`urls.0: Each URL may be at most 8,192 characters.`).
## Response
| Field | Type | Description |
| --- | --- | --- |
| `requestId` | `string` | Id of this request, the same for every repeat with the same Idempotency-Key. Also returned as X-Request-Id. |
| `results[]` | `object[]` | One entry per successfully fetched URL. |
| `results[].url` | `string` | Final URL after redirects. |
| `results[].title` | `string \| null` | Page title. |
| `results[].author / publishedDate / image` | `string \| null` | Metadata when the page exposes it. |
| `results[].favicon` | `string \| null` | Favicon URL. |
| `results[].markdown` | `string` | Main content as Markdown. |
| `results[].text` | `string` | Plain text, tags removed, whitespace normalized. |
| `results[].html` | `string` | Page HTML as served. Only with html: true, and only while the response stays within its 24 MiB size limit (else warning RESPONSE_SIZE_LIMIT). |
| `results[].textTruncated` | `boolean` | true when text or markdown was cut at text.maxCharacters, or at the response size limit (warning RESPONSE_SIZE_LIMIT). Present whenever text is enabled. |
| `results[].warnings` | `{ code, message }[]` | Absent when empty. "PAGE_TRUNCATED" = only part of the page could be retrieved; content after that point is missing. "THIN_CONTENT" = very little text, the page probably needs render: true. "EXTRACTION_FALLBACK" = the page was too complex to convert in time; text is plain and markdown has no structure. "RESPONSE_SIZE_LIMIT" = the response reached its 24 MiB size limit: this result has no html, or no html and cut text/markdown. |
| `statuses[]` | `object[]` | One entry per requested URL, success or error. |
| `statuses[].status` | `"success" \| "error"` | Outcome for this URL. |
| `statuses[].source` | `"livecrawled" \| null` | Always "livecrawled" on success: every request is fetched live. null on error. |
| `statuses[].fetchedAt` | `string \| null` | When the page was fetched (ISO 8601). null on error. |
| `statuses[].strategy` | `"fast" \| "retry" \| "browser"` | Fetch strategy. "fast" = standard page (1 credit); "retry" = the first attempt did not return the page, a retry through another connection did (2 credits); "browser" = rendered in a real browser, as a normal visitor sees it (3 credits, 5 when it needed a retry through another connection). |
| `statuses[].credits` | `integer` | Credits charged for this URL: 1 standard, 2 retried, 3 rendered, 5 rendered + retried, 0 on error and for a short page (thin). |
| `statuses[].fallbackUsed` | `boolean` | Present (true) only when the browser fallback ran for this URL. With strategy "browser" the content is that render (fast → render); otherwise rendering was tried and did not help. |
| `statuses[].thin` | `boolean` | Present (true) only for a short page: fewer than 500 characters of visible text even after rendering. Delivered as a success at 0 credits, with only that visible text. |
| `statuses[].error` | `{ tag, message } \| null` | Per-URL error, see crawl errors. |
| `searchTime` | `integer (ms)` | Total processing time. |
Browser fallback is on by default: a page the fast fetch cannot read (no or very little visible text, blocked, 401/403, a timeout) is rendered in a real browser and charged only the render credits. `statuses[].fallbackUsed` is `true` when that happened (`strategy: "browser"` = the content is the render). Send `renderFallback: false` for a fast fetch only. A page that still shows fewer than 500 characters of visible text after rendering is delivered as a success with `statuses[].thin: true` at 0 credits, holding only that visible text — it is not an error.
`text.maxCharacters` (default 100,000, max 1,000,000) cuts `text` and `markdown`; `results[].textTruncated` is `true` when it did. A response carries at most 24 MiB of JSON: from the result that would pass it on, results come without `html`, and if that is not enough, `text` and `markdown` are cut (`textTruncated: true`), each such result flagged `RESPONSE_SIZE_LIMIT`. The request never fails because of it. `title` and `author` are at most 2,000 characters each; an `image` or `favicon` URL longer than 4,096 characters is left out (`favicon` then falls back to `/favicon.ico`). The MCP tool `get_page_contents` has its own, smaller default of 8,000 (`max_characters`), because its result goes into an agent's context. `results[].warnings` is present only when there is something to know about a page:
| Code | Meaning | What to do |
| --- | --- | --- |
| `PAGE_TRUNCATED` | Only part of the page could be retrieved (size limit or an interrupted transfer); content after that point is missing. | Nothing to retry; the result is the part that arrived. |
| `THIN_CONTENT` | Very little text next to scripts or an empty app root: the page probably loads its content with JavaScript. The URL is still charged as a success. | Retry with `render: true`. The default browser fallback renders a page with fewer than 500 characters of visible text by itself, not one with more. |
| `EXTRACTION_FALLBACK` | The page structure was too complex to convert in time; `text` is plain and `markdown` carries no structure. | Use `text`. |
| `RESPONSE_SIZE_LIMIT` | The response reached its 24 MiB size limit. This result and every later one come without `html`; when that was not enough, `text` and `markdown` were cut too (`textTruncated: true`). The URL is still a success and is charged. | Request fewer URLs at once, or without `html`. |
## Response headers
| Header | Description |
| --- | --- |
| `X-Request-Id` | Same as requestId in the body. |
| `X-Credits-Remaining` | Credit balance after this request. |
| `X-RateLimit-Limit` | URLs you can send at once (bucket size, = rateLimit.burst in GET /v1/account). The bucket refills at rateLimit.requestsPerSecond URLs per second. |
| `X-RateLimit-Remaining` | URLs you can send right now. |
| `Retry-After` | Seconds to wait: before retrying a 429, or before repeating a 202 with the same Idempotency-Key. |
| `Idempotency-Key` | The request's Idempotency-Key: yours, or one the API created when you sent none. On every answer (200, 202, errors) except a 400 for a malformed key and a 413 for a body over 10 MiB (refused unread). Repeat a 202 with it. |
## Errors
### Per-URL errors
Returned inside `statuses[].error` with HTTP 200. These never cost credits.
| Tag | Meaning |
| --- | --- |
| `CRAWL_NOT_FOUND` | The page returned 404/410. |
| `SOURCE_NOT_AVAILABLE` | The target refused access (e.g. 403 or 429) after retries. |
| `CRAWL_LIVECRAWL_TIMEOUT` | The live fetch did not finish within livecrawlTimeout. |
| `CRAWL_NOT_RENDERED` | No usable content, e.g. the page needs JavaScript rendering (retry without renderFallback: false, or with render: true), or a rendered page shows no visible text. 0 credits. |
| `UNSUPPORTED_CONTENT_TYPE` | The URL does not serve HTML or text. |
| `CRAWL_UNKNOWN_ERROR` | Something else went wrong. Safe to retry. |
| `SOURCE_BLOCKED` | Access denied by the site, or it answered with an interstitial page (HTTP 202) instead of the content. 0 credits. |
| `BROWSER_UNAVAILABLE` | Browser rendering is temporarily unavailable; no credits were charged. |
| `RATE_LIMITED` | Rendering limit for your plan reached; no credits were charged. |
| `RESTRICTED_TARGET` | This URL is restricted. Other URLs in the same request are still processed. 0 credits. |
### Request errors
Returned with a non-200 status and no results.
| HTTP | Tag | Meaning |
| --- | --- | --- |
| 400 | `INVALID_REQUEST` | Invalid parameters. |
| 400 | `TOO_MANY_URLS` | More URLs than allowed per request (10). |
| 401 | `UNAUTHORIZED` | Missing or invalid API key. |
| 402 | `INSUFFICIENT_CREDITS` | Not enough credits. Top up or upgrade. |
| 403 | `KEY_SUSPENDED` | The key or account is suspended or not yet approved. |
| 429 | `RATE_LIMITED` | Rate limit exceeded. Retry after the Retry-After header. |
| 409 | `IDEMPOTENCY_CONFLICT` | The Idempotency-Key was already used with a different request, or its result is no longer available (results are kept 15 minutes). Use a new key. |
| 413 | `PAYLOAD_TOO_LARGE` | The request body is larger than 10 MiB. Nothing was read or charged. |
| 451 | `RESTRICTED_TARGET` | All requested URLs are restricted. No credits charged. |
| 503 | `SERVICE_UNAVAILABLE` | Temporarily unavailable. Retry shortly. |
```json title="Error body"
{
"error": {
"code": 402,
"tag": "INSUFFICIENT_CREDITS",
"message": "Not enough credits. Top up or upgrade your plan."
},
"requestId": "req_01J…"
}
```
## Long requests and Idempotency-Key
Most requests answer within seconds. A request still fetching after about 100 seconds (slow sites, rendering, a long `livecrawlTimeout`) answers `202` instead of making you wait on an open connection:
```json title="202 Accepted"
{
"status": "pending",
"requestId": "req_01J…",
"idempotencyKey": "lwda_3f1c2b7e-0d4a-4c1e-9a55-2b8f0e6c7d10",
"retryAfter": 1,
"message": "Still fetching. Send the same request again with the same Idempotency-Key header to get the result; it is not charged again."
}
```
Send the **same request body** again with the header `Idempotency-Key: ` after `Retry-After` seconds. The repeat waits for the fetch and answers `200` with the result, or `202` again if it is still running. A repeat is never charged again. The SDKs and the MCP server do this for you.
**Always send your own `Idempotency-Key`** (8–128 characters of `A–Z a–z 0–9 _ - : .`, for example a UUID) on the first request, and keep it until you have the answer. A connection can break after the API received the request: a network error, a timeout on your side, a deploy. Send the same request again with the same key, and you get that fetch (its result, or `202` while it still runs) instead of a new one, so you pay once. Without your own key, a cut connection is a lost answer: the retry has no key, so it is a new request and charged again.
Every answer of `POST /v1/contents` (`200`, `202` and errors) carries the request's key in the `Idempotency-Key` response header: yours, or one the API created when you sent none. The only exceptions are the `400` for a malformed key and the `413` for a request body over 10 MiB, which is refused before it is read (nothing was charged, there is nothing to repeat). The SDKs create a key for each `contents()` call when you pass none, send it on every retry and put it on the error they raise (`idempotencyKey` / `idempotency_key`).
- Results are kept for **15 minutes**. A result nobody collects stays charged; after 15 minutes a repeat answers `409 IDEMPOTENCY_CONFLICT`, and a new key fetches the pages again.
- A key is bound to its request body for 24 hours: reusing it with a different body answers `409 IDEMPOTENCY_CONFLICT`. Use a new key for every new request.
## Rate limits
Limits are per account and count **URLs per second** (a request with 3 URLs uses 3). When exceeded, the API returns `429` with a `Retry-After` header (seconds). Limits work as a bucket: `X-RateLimit-Limit` is how many URLs you can send at once (`rateLimit.burst` in `GET /v1/account`), `X-RateLimit-Remaining` how many right now, and the bucket refills at your plan's URLs per second (`rateLimit.requestsPerSecond`).
| Plan | Price / month | Rate limit (URLs/s) |
| --- | --- | --- |
| Lite | $10 | 1 |
| Starter | $100 | 2 |
| Developer | $500 | 5 |
| Business | $900 | 10 |
| Enterprise | $2,250 | 20 |
| Custom | Custom | Custom |
## Credits
| Request | Credits |
| --- | --- |
| Standard page (static / server-rendered) | 1 |
| Retried page (succeeded on a retry through another connection) | 2 |
| JS-rendered page | 3 |
| Rendered + retried page | 5 |
| Short page (under 500 characters of visible text; delivered as that text, a success) | 0 |
| Failed request (incl. access denied by the site) | 0 |
- Plan credits reset monthly.
- Top-up credits never expire.
When your balance is too low for the URLs in a request, the API returns `402 INSUFFICIENT_CREDITS`. The current balance is returned in `X-Credits-Remaining` on every response.
## Account
`GET https://livewebdata.com/api/v1/account` returns your plan, remaining credits and rate limit. It is free — call it before large jobs.
```json title="GET /v1/account"
{
"plan": "starter",
"status": "active",
"credits": { "plan": 81234, "topup": 10000, "total": 91234 },
"rateLimit": { "requestsPerSecond": 2, "burst": 4 },
"periodEndsAt": "2026-11-01T00:00:00Z"
}
```
---
# Examples
Copy-paste examples in cURL, Python and JavaScript. Replace YOUR_API_KEY with a key from the dashboard.
## Basic fetch
The smallest possible request — one URL, defaults for everything else.
```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"urls": [
"https://example.com/article"
]
}'
```
```python title="Python"
import requests
response = requests.post(
"https://livewebdata.com/api/v1/contents",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json={
"urls": ["https://example.com/article"]
},
timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```
```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
method: "POST",
headers: {
Authorization: "Bearer YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({
"urls": [
"https://example.com/article"
]
}),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```
## Geotargeting
Fetch a page as seen from Japan.
```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"urls": [
"https://example.com/pricing"
],
"countryCode": "jp"
}'
```
```python title="Python"
import requests
response = requests.post(
"https://livewebdata.com/api/v1/contents",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json={
"urls": ["https://example.com/pricing"],
"countryCode": "jp"
},
timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```
```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
method: "POST",
headers: {
Authorization: "Bearer YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({
"urls": [
"https://example.com/pricing"
],
"countryCode": "jp"
}),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```
## Text limits
Return at most 5,000 characters of the full page body, including navigation and footer.
```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"urls": [
"https://example.com/docs/getting-started"
],
"text": {
"maxCharacters": 5000,
"verbosity": "full"
}
}'
```
```python title="Python"
import requests
response = requests.post(
"https://livewebdata.com/api/v1/contents",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json={
"urls": ["https://example.com/docs/getting-started"],
"text": {
"maxCharacters": 5000,
"verbosity": "full"
}
},
timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```
```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
method: "POST",
headers: {
Authorization: "Bearer YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({
"urls": [
"https://example.com/docs/getting-started"
],
"text": {
"maxCharacters": 5000,
"verbosity": "full"
}
}),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```
## Several URLs
Fetch up to 10 pages in one request. Each URL gets its own status; only successful pages cost credits.
```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"urls": [
"https://example.com/a",
"https://example.com/b",
"https://example.com/blocked"
]
}'
```
```python title="Python"
import requests
response = requests.post(
"https://livewebdata.com/api/v1/contents",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json={
"urls": ["https://example.com/a", "https://example.com/b", "https://example.com/blocked"]
},
timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```
```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
method: "POST",
headers: {
Authorization: "Bearer YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({
"urls": [
"https://example.com/a",
"https://example.com/b",
"https://example.com/blocked"
]
}),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```
## Whole jobs
For URL lists, retries and scheduling see [Automate requests](https://livewebdata.com/docs/automation) — including ready-made Python and Node scripts.
---
# Automate requests
Move from clicking Run in the playground to fetching URL lists from your own code — with rate limits, retries and scheduling handled.
## From playground to code
Every run in the [playground](https://livewebdata.com/dashboard/playground) shows a **Use it in code** panel with cURL, Python, JavaScript and SDK snippets for exactly the request you just made. Start there, then move to the scripts below.
Download all example scripts: https://livewebdata.com/live-web-data-api-examples.zip
| File | What it does |
| --- | --- |
| `examples/python/quickstart.py` | One request, prints title + first 500 chars of markdown |
| `examples/python/batch_from_csv.py` | Reads `urls.csv` (`url`, optional `country`), fetches with concurrency = rate limit, retries with backoff, writes `results.jsonl` and prints a summary |
| `examples/node/quickstart.mjs` | Same as the Python quickstart, native `fetch`, no dependencies |
| `examples/node/batch_from_csv.mjs` | Same as the Python batch script, no dependencies |
| `examples/urls.csv` | 10 sample URLs incl. a restricted, a missing and a timing-out URL to show error handling |
```bash title="Run the batch script"
export LIVE_WEB_DATA_API_KEY=1337_live_… # never hard-code keys
python examples/python/batch_from_csv.py examples/urls.csv
node examples/node/batch_from_csv.mjs examples/urls.csv
```
## Process a list of URLs
The simplest loop: one request per URL, one after the other.
```python title="Python"
import os, requests
API = os.environ.get("LIVE_WEB_DATA_API_BASE_URL", "https://livewebdata.com/api")
headers = {"Authorization": f"Bearer {os.environ['LIVE_WEB_DATA_API_KEY']}"}
for url in ["https://example.com/a", "https://example.com/b"]:
r = requests.post(f"{API}/v1/contents", json={"urls": [url]}, headers=headers, timeout=60)
status = r.json()["statuses"][0]
print(url, status["status"], status["error"] and status["error"]["tag"])
```
```javascript title="Node"
const API = process.env.LIVE_WEB_DATA_API_BASE_URL ?? "https://livewebdata.com/api";
const headers = { Authorization: `Bearer ${process.env.LIVE_WEB_DATA_API_KEY}`, "Content-Type": "application/json" };
for (const url of ["https://example.com/a", "https://example.com/b"]) {
const res = await fetch(`${API}/v1/contents`, { method: "POST", headers, body: JSON.stringify({ urls: [url] }) });
const { statuses } = await res.json();
console.log(url, statuses[0].status, statuses[0].error?.tag ?? "");
}
```
You can also send up to 10 URLs per request — see [Several URLs](https://livewebdata.com/docs/examples#several-urls).
## Respect rate limits
- Keep a steady pace of **your plan's URLs per second** (from `GET /v1/account` → `rateLimit.requestsPerSecond`). Bursts up to `rateLimit.burst` URLs (the bucket size, also sent as `X-RateLimit-Limit`) are fine; the bucket refills at that rate.
- Watch `X-RateLimit-Remaining` on each response: it is how many URLs you can send right now.
- On `429`, wait `Retry-After` seconds and retry the same request.
## Retries
Retry `429`, `5xx` and per-URL `CRAWL_LIVECRAWL_TIMEOUT` up to **3 times** with exponential backoff and jitter (e.g. `0.5s · 2^n + random(0–0.25s)`). Do **not** retry:
| Outcome | Why |
| --- | --- |
| `451 RESTRICTED_TARGET` | The domain is blocked; repeated attempts can suspend your key |
| `400 INVALID_REQUEST` / `TOO_MANY_URLS` | Fix the request first |
| `401` / `403` | Key invalid, revoked or suspended |
| `402 INSUFFICIENT_CREDITS` | Stop the job cleanly, top up, resume |
| `CRAWL_NOT_FOUND` | The page doesn't exist |
**Send your own `Idempotency-Key`** with every `POST /v1/contents` (one per logical request, for example per URL of your list) and send the same key on every retry of that request. A connection cut after the API received the request (a network error, a timeout, a deploy) is then retried without a second charge: the repeat returns that fetch's result. A `202` (still fetching) is repeated the same way, after `Retry-After`. Use a **new** key for a new attempt at a URL that already has an answer, such as a per-URL `CRAWL_LIVECRAWL_TIMEOUT`: the same key returns the same answer. See [Long requests and Idempotency-Key](https://livewebdata.com/docs/api-reference#long-requests-and-idempotency-key).
The [SDKs](https://livewebdata.com/docs/sdks) implement this policy for you.
## Check your balance first
Call `GET /v1/account` before a large job and compare `credits.total` with the number of URLs. Stop cleanly when a request returns `402`.
```bash title="cURL"
curl https://livewebdata.com/api/v1/account -H "Authorization: Bearer $LIVE_WEB_DATA_API_KEY"
```
## Scheduling
Any scheduler works — the scripts are plain processes that exit when done.
```bash title="cron (daily at 06:00)"
0 6 * * * cd /opt/jobs && LIVE_WEB_DATA_API_KEY=… /usr/bin/python3 batch_from_csv.py urls.csv --out results.jsonl
```
```yaml title=".github/workflows/fetch.yml"
name: Fetch pages
on:
schedule: [{ cron: "0 6 * * *" }]
workflow_dispatch:
jobs:
fetch:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: node examples/node/batch_from_csv.mjs examples/urls.csv --out results.jsonl
env:
LIVE_WEB_DATA_API_KEY: ${{ secrets.LIVE_WEB_DATA_API_KEY }}
```
Other schedulers (Cloud Scheduler, Kubernetes CronJob, n8n, Airflow) run the same command.
## Keep your key safe
- Store it in an environment variable (`LIVE_WEB_DATA_API_KEY`) or your platform's secret manager.
- Never put it in client-side code, mobile apps, browser extensions or repositories.
- Use one key per environment (e.g. production, staging, n8n) — you can label keys in the dashboard and see the label in the usage log.
- Rotate a key immediately if it leaks: create a new one, then revoke the old one.
---
# SDKs
Thin, typed clients for JavaScript/TypeScript and Python — built-in retries, typed errors, one-line page fetches.
## Install
The SDKs live in the repository under `packages/` and are not published yet.
```bash title="JavaScript / TypeScript"
cd packages/sdk-js && npm install && npm run build
npm install ./packages/sdk-js # in your project
```
```bash title="Python"
pip install ./packages/sdk-python # requires httpx
```
## JavaScript / TypeScript
```javascript title="JS SDK"
import { Teneo } from "@teneo/contents";
const client = new Teneo({ apiKey: process.env.LIVE_WEB_DATA_API_KEY, baseUrl: "https://livewebdata.com/api" });
const page = await client.getPage("https://example.com/article", { country: "de", format: "markdown" });
console.log(page.title, page.credits);
const res = await client.contents({ urls: ["https://example.com/a", "https://example.com/b"] });
const account = await client.account();
```
## Python
```python title="Python SDK"
import os
from teneo import Teneo
client = Teneo(api_key=os.environ["LIVE_WEB_DATA_API_KEY"], base_url="https://livewebdata.com/api")
page = client.get_page("https://example.com/article", country="de", format="markdown")
print(page.title, page.credits)
res = client.contents(urls=["https://example.com/a", "https://example.com/b"])
account = client.account()
```
Both clients read the key from `LIVE_WEB_DATA_API_KEY` and the base URL from `LIVE_WEB_DATA_API_BASE_URL` when you don't pass them.
## Methods
| JS | Python | Description |
| --- | --- | --- |
| `contents(request, { idempotencyKey })` | `contents(urls=…, country_code=…, text=…, html=…, livecrawl_timeout=…, render=…, render_fallback=…, idempotency_key=…)` | Raw `POST /v1/contents` (1–10 URLs). `render_fallback=False` / `renderFallback: false` turns the default browser fallback off |
| `getPage(url, opts)` | `get_page(url, country=…, format=…)` | One URL → `{ title, url, content, credits, source }`; raises if that URL failed. `format: "html"` requests the HTML (`html: true`) |
| `account()` | `account()` | `GET /v1/account` |
## Errors
| Error | When |
| --- | --- |
| `TeneoAuthError` | 401 / 403 — key invalid, revoked or suspended |
| `TeneoInsufficientCreditsError` | 402 |
| `TeneoRateLimitError` | 429 after all retries (`retryAfter` / `retry_after`) |
| `TeneoIdempotencyError` | 409 — the idempotency key was used with a different request, or its result expired (15 minutes) |
| `TeneoRestrictedTargetError` | 451 |
| `TeneoConnectionError` | Network error or timeout after all retries, or a `2xx` answer cut while it was read. The API may have received the request: repeat it with the error's `idempotencyKey` / `idempotency_key` |
| `TeneoPendingError` | Still `202` when `maxWaitMs` / `max_wait` ran out; `idempotencyKey` / `idempotency_key` collects the result later |
| `TeneoPageError` | `getPage` / `get_page`: the URL failed (`tag`, e.g. `CRAWL_NOT_FOUND`) |
| `TeneoAPIError` | Base class (status, tag, requestId, and `idempotencyKey` / `idempotency_key`: the key of a `contents()` call, else `null` / `None`) |
## Retries
Both SDKs retry `429` (respecting `Retry-After`), `5xx`, network errors, a `2xx` answer cut while it is read and — for `getPage` — `CRAWL_LIVECRAWL_TIMEOUT`, up to 3 times with exponential backoff and jitter. `400`, `401`, `402`, `403`, `409` and `451` are never retried. Configure with `maxRetries` / `max_retries`.
Every attempt of one `contents()` call sends the same `Idempotency-Key`, so a retry picks up the first fetch and is never charged twice (render requests included). A fetch that runs longer than about 100 seconds answers `202`; the SDKs repeat it after `Retry-After` until the result arrives, for up to 5 minutes (`maxWaitMs` / `max_wait`). These repeats do not count as retries. See [Long requests](https://livewebdata.com/docs/api-reference#long-requests-and-idempotency-key).
Without your own key, the SDKs create one per `contents()` call. When the retries run out, the error carries that key: call `contents()` again with the same request and `{ idempotencyKey: e.idempotencyKey }` / `idempotency_key=e.idempotency_key` within 15 minutes, and a fetch the API already received returns its result instead of being charged again. **Better: always pass your own key** (one per logical request, for example per row of your input), so a request cut by a network error, a timeout or a deploy can be retried without a second charge even after your process restarts.
---
# MCP for agents
Use the Live Web Data API as a tool in Claude, Cursor, VS Code and any other MCP client — with one URL and your existing API key.
## What is MCP
The Model Context Protocol (MCP) is an open standard that lets AI agents discover and call external tools. Our MCP server exposes the Contents API as two tools, so an agent can read any public web page on its own.
## Server & auth
| | Value |
| --- | --- |
| Server URL | `https://livewebdata.com/mcp` |
| Transport | Streamable HTTP (stateless): JSON-RPC over `POST`; `GET` and `DELETE` answer `405` |
| Auth | `Authorization: Bearer ` |
| Keys | Same keys as the REST API — create one under [API keys](https://livewebdata.com/dashboard/keys) |
Missing or invalid keys are rejected with HTTP 401 (suspended accounts: 403, "API access for this account is suspended. Contact support.") before any tool runs.
## Client setup
Replace `YOUR_KEY` with your API key. In the dashboard, the [Integrations](https://livewebdata.com/dashboard/integrations) page fills in your key for you.
### Claude Code
Run this in your terminal. The server is then available in every Claude Code session for this project.
```bash
claude mcp add --transport http live-web-data https://livewebdata.com/mcp \
--header "Authorization: Bearer YOUR_KEY"
```
Add `--scope user` to make it available in all projects. Check with `claude mcp list`.
### Cursor
Add this to your Cursor MCP config, globally or per project.
File: `~/.cursor/mcp.json or .cursor/mcp.json`
```json
{
"mcpServers": {
"live-web-data": {
"url": "https://livewebdata.com/mcp",
"headers": {
"Authorization": "Bearer YOUR_KEY"
}
}
}
}
```
Prefer an env var: `"Bearer ${env:LIVE_WEB_DATA_API_KEY}"`.
### VS Code
Add this to your workspace. VS Code asks for the key the first time the server starts and stores it securely.
File: `.vscode/mcp.json`
```json
{
"inputs": [
{
"type": "promptString",
"id": "live-web-data-key",
"description": "Live Web Data API key",
"password": true
}
],
"servers": {
"live-web-data": {
"type": "http",
"url": "https://livewebdata.com/mcp",
"headers": {
"Authorization": "Bearer ${input:live-web-data-key}"
}
}
}
}
```
Uses the top-level key `servers` (not `mcpServers`). Paste `YOUR_KEY` when prompted.
### Claude Desktop
Claude Desktop connects to remote servers through the mcp-remote bridge. Add this to your Claude Desktop config and restart the app.
File: `claude_desktop_config.json`
```json
{
"mcpServers": {
"live-web-data": {
"command": "npx",
"args": [
"-y",
"mcp-remote",
"https://livewebdata.com/mcp",
"--header",
"Authorization:${AUTH_HEADER}",
"--transport",
"http-only"
],
"env": {
"AUTH_HEADER": "Bearer YOUR_KEY"
}
}
}
}
```
Requires Node.js. Alternative on paid plans: Settings → Connectors → Add custom connector with request header `authorization` = `Bearer …` (where available; the server must be publicly reachable).
### Other MCP clients
Any MCP client that supports Streamable HTTP can connect directly with the URL and the Authorization header. For stdio-only clients, use the mcp-remote bridge:
```bash
npx -y mcp-remote https://livewebdata.com/mcp \
--header "Authorization: Bearer YOUR_KEY" --transport http-only
```
Server URL: `https://livewebdata.com/mcp` · Transport: Streamable HTTP · Auth header: `Authorization: Bearer `
### REST API
Prefer plain HTTP? The same API key works with the REST endpoint.
```bash
curl -X POST https://livewebdata.com/api/v1/contents \
-H "Authorization: Bearer YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{"urls": ["https://example.com/article"]}'
```
## get_page_contents
Fetch a public web page live and return its main content as clean Markdown, plain text or HTML. Read-only, idempotent. Single URL per call.
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| `url` | string (http/https, at most 8,192 characters) | required | The public URL to fetch. |
| `country` | ISO-3166 alpha-2 | `"us"` | Fetch the page as seen from this country, e.g. 'de', 'gb', 'us'. |
| `format` | `"markdown"` \| `"text"` \| `"html"` | `"markdown"` | Output format. Prefer markdown. The page HTML is fetched and returned only for `"html"`. |
| `max_characters` | integer 500–100000 | `8000` | Maximum characters of content to return. Smaller than the REST API's default (`text.maxCharacters`, 100,000) because the result goes into the agent's context; raise it when the content is cut. |
| `verbosity` | `"compact"` \| `"full"` | `"compact"` | compact = main content only, full = whole page body. |
The result is one text item with a short header, followed by the content. The full REST-style JSON is also returned as `structuredContent` (with `results[].html` only for `format: "html"`).
```markdown title="Tool result"
# Example article
Source: https://example.com/article · country: us · live · credits used: 1 · credits left: 8,411
# Example article
Main content as clean Markdown…
```
## get_account_status
Check the remaining credits, plan and rate limit for the connected account. Agents should call it before large batches. No parameters, free. Returns the same numbers as `GET /v1/account`.
```text title="Tool result"
Plan: Starter · Credits left: 8,412 · Rate limit: 2 URL/s (burst 4) · Status: active
```
## Errors
Errors are returned with `isError: true` and one actionable sentence the agent can follow. When nothing was charged, the message ends with “No credits were charged.”
| Case | Message to the agent |
| --- | --- |
| 451 restricted target | This website is restricted and cannot be retrieved. Do not retry; use another source. |
| `CRAWL_NOT_RENDERED` | The page has no usable text: it needs JavaScript rendering, or shows no text even when rendered. Try another source or a different URL for the same information. |
| `CRAWL_NOT_FOUND` | Page not found (404/410). |
| `SOURCE_NOT_AVAILABLE` | The website refused this request. Retry once with a different country, otherwise use another source. |
| `SOURCE_BLOCKED` | Access denied by the website. Retry later or with a different country, otherwise use another source. |
| `CRAWL_LIVECRAWL_TIMEOUT` | The page timed out. Retry once with a longer timeout, otherwise use another source. |
| `BROWSER_UNAVAILABLE` | Browser rendering is temporarily unavailable. Retry in a few minutes, otherwise use another source. |
| `RATE_LIMITED` | The rendering limit for the user's plan is reached. Do not retry rendering now; use another source. |
| 402 insufficient credits | The account is out of credits. Tell the user to top up at …/dashboard/billing. |
| 429 rate limited | Rate limit reached. Wait N seconds before retrying. |
| 401 / 403 | The API key is invalid or suspended. Tell the user to check their key at …/dashboard/keys. |
| Still fetching after 5 minutes | The page did not finish loading within 5 minutes, and the fetch may still be charged. Use another source. |
## Credits
MCP calls are billed exactly like REST calls: 1 credit per successful page, failed requests (including access denied by the site) are free. They appear in Usage with channel `mcp`. See [Credits](https://livewebdata.com/docs/api-reference#credits).
---
# No-code recipes
Call the API from n8n, Make or Zapier with each tool's generic HTTP step. Native integrations are on the roadmap.
## Before you start
You need an API key from [API keys](https://livewebdata.com/dashboard/keys). Every recipe sends the same request:
| Setting | Value |
| --- | --- |
| Method | `POST` |
| URL | `https://livewebdata.com/api/v1/contents` |
| Header | `Authorization: Bearer YOUR_KEY` |
| Body (JSON) | `{"urls": ["https://example.com/article"], "countryCode": "us"}` |
The response is JSON: page content in `results[0].markdown`, the outcome in `statuses[0].status` and `statuses[0].error.tag`.
## n8n
> **Native integration coming soon.** Use the built-in **HTTP Request** node for now.
_Screenshot coming soon: n8n HTTP Request node_
| Field (HTTP Request node) | Value |
| --- | --- |
| Method | `POST` |
| URL | `https://livewebdata.com/api/v1/contents` |
| Authentication | Generic Credential Type → **Bearer Auth** (token = your key) — or **Header Auth** with Name `Authorization`, Value `Bearer YOUR_KEY` |
| Send Body | on |
| Body Content Type | `JSON` |
| Specify Body | `Using JSON` → `{"urls": ["{{ $json.url }}"]}` |
| Options → Response → Response Format | `JSON` (default: autodetect) |
**URL lists:** feed one item per URL into the node. Use the node's **Options → Batching** (*Items per Batch* = your rate limit, *Batch Interval (ms)* = 1000) — or a **Loop Over Items** node plus a **Wait** node (1 second) to stay within your rate limit.
## Make
> **Native integration coming soon.** Use the **HTTP** app's **Make a request** module for now.
_Screenshot coming soon: Make HTTP "Make a request" module_
| Field (HTTP → Make a request) | Value |
| --- | --- |
| Authentication type | `API key` (stores the key in a keychain) — or `No authentication` plus the header below |
| URL | `https://livewebdata.com/api/v1/contents` |
| Method | `POST` |
| Headers | `Authorization` = `Bearer YOUR_KEY` |
| Body content type | `application/JSON` |
| Body input method | `JSON string` |
| Body content | `{"urls": ["https://example.com/article"]}` |
| Parse response | `Yes` |
**URL lists:** use an **Iterator** over your URLs before the HTTP module and set the scenario's scheduling so the number of operations per second stays within your rate limit.
## Zapier
> **Native integration coming soon.** Use **Webhooks by Zapier** → **Custom Request** for now (not available on the Free plan).
_Screenshot coming soon: Zapier Webhooks by Zapier – Custom Request_
| Field (Custom Request) | Value |
| --- | --- |
| Method | `POST` |
| URL | `https://livewebdata.com/api/v1/contents` |
| Data Pass-Through? | `False` |
| Data | `{"urls": ["https://example.com/article"]}` |
| Headers | `Authorization` = `Bearer YOUR_KEY`, `Content-Type` = `application/json` |
The JSON response is available as fields in later steps.
## Tips
- Failed pages return HTTP 200 with `statuses[0].status = "error"` — add a filter/router on that field.
- Keep the key in the tool's credential store, not in plain step fields, wherever possible.
---