Skip to content

Pages by country

Which language did you fetch? Accept-Language and Content-Language

Learn what Accept-Language asks for, what Content-Language and the html lang attribute report, and how to record the language of a fetched page for the model.

By Curtis Mccann, Live Web Data team · Published

This article was drafted with AI assistance from the sources listed below and edited and published by the Live Web Data team. Disclosure: the Live Web Data API, which this article mentions, is this site's own product.

Record the language of every page you fetch
  • What you asked for: the Accept-Language value you sent, or none.
  • Where you ended up: the final URL after redirects, for example a /de/ path.
  • What the page says: the lang attribute on the html element.
  • What the header says: Content-Language, read as the intended audience.
  • What the text is: a look at the text itself when the signals disagree.

You ask for a page, you get text back, and you pass it to the model. Somewhere in that chain the site picked a language for you. Sometimes it is the one you wanted. Sometimes it is the site's fallback, a redirect to another locale, or a page that only gets translated after scripts run in a browser. If you do not record which language you got, the model cannot tell either, and a German page summarized for an English-speaking user can come out as a confident mistranslation.

This post covers the request side (what Accept-Language asks for), the response side (what Content-Language and the lang attribute report), and how to carry the result into the prompt.

What Accept-Language asks for

The decision first: send Accept-Language when you know the language the answer needs, and leave it out on purpose when you do not.

RFC 9110 defines the header as a list of language ranges with optional quality values. Its own example is Accept-Language: da, en-gb;q=0.8, en;q=0.7, which it reads as "I prefer Danish, but will accept British English and other types of English".

Two rules from the same RFC matter for a fetcher:

  • No header means no preference. RFC 9110 says "a request that does not contain the field implies that the sender has no preference on that dimension of negotiation." The site is free to serve whatever version it picks for such requests.
  • The server may ignore it. When no available version fits the preference, "the origin server can either honor the header field by sending a 406 (Not Acceptable) response or disregard the header field". A header is a request, not a switch.

The RFC also warns about privacy: "It might be contrary to the privacy expectations of the user to send an Accept-Language header field" with a user's complete language preferences on every request. For an AI app, send the language the answer needs, not the user's full browser list.

When the header changes nothing

Some sites never look at the header. Some choose the language in the browser after the page loads. The guide on web pages by country covers how the country a page is fetched from can change what you get.

The browser case is easy to miss. A bug report filed with Wikimedia in February 2021, titled "Portal ignores Accept-Language header, content always in English language", was closed as invalid. The developer's explanation: the portal has no server-side component, and translations are applied in the browser, so the server sends one static version to everyone. A fetcher that sends a different Accept-Language gets that same version. The lesson is general: when the header seems ignored, the site may be choosing the language somewhere your fetcher never reaches.

Three signals for the language you got

The decision: read three signals from every response, and trust the page's own markup over the header.

1. The final URL

A redirect is the clearest signal. If you asked for https://example.com/pricing and ended up on https://example.com/de/pricing, the site chose a German version for you. Keep both addresses; the post on final URL vs canonical URL explains which one to cite.

2. The lang attribute

The W3C Internationalization article on HTTP and language says "you should always use a lang attribute on the html tag" to declare the page's default language, and that "the language attribute always overrides the HTTP header information." So <html lang="de"> is the page's own statement about its text. MDN agrees: to indicate the language a document is written in, use the lang attribute.

3. Content-Language, read with care

RFC 9110 says Content-Language "describes the natural language(s) of the intended audience for the representation", and adds: "Note that this might not be equivalent to all the languages used within the representation." MDN gives an example: Content-Language: de-DE marks a document for German speakers that may be written in English, as part of a language course. The header can also list several audiences, as in the RFC's bilingual Maori and English example, Content-Language: mi, en. And "If no Content-Language is specified, the default is that the content is intended for all language audiences."

The W3C article says the header "should be used only to provide metadata about the intended audience." Treat it as a hint about who the page is for, not as the language of the text.

A Vary header helps too. RFC 9110's example Vary: accept-encoding, accept-language says the server may have used those request headers, or their absence, to choose the response. If accept-language is listed, the header you sent may have mattered. If it is missing, the site can still choose a language in another way.

Check it from the command line

These commands run as written against a public page. The first sends a German preference and prints the redirect target and the language headers. The second prints the lang attribute from the HTML.

curl -s -o /dev/null -D - -H "Accept-Language: de-DE,de;q=0.9" https://example.com/ \
  | grep -iE '^(location|content-language|vary):'

curl -s -H "Accept-Language: de-DE,de;q=0.9" https://example.com/ \
  | grep -oiE '<html[^>]*lang="[^"]*"' | head -n 1

Run both again without the -H option. If the output is the same, this page does not switch language on the header.

When the signals disagree, for example lang="en" on a page whose text is plainly German, look at the text itself. Markup is written by people and templates, and it can be wrong.

Pass the language to the model

Store the language you detected next to the URL, the country and the fetch time, and put it in the source label:

<source url="https://example.com/de/pricing" requested="https://example.com/pricing" lang="de" country="de" fetched="2026-11-21T08:00:00Z">
...page text...
</source>
Answer in the user's language. If the source is in another language, say so and translate only what you quote.

This keeps the model from presenting a translated summary as if the page itself were in the user's language, and it lets you spot sources that keep returning the wrong one.

Public pages, by the rules

Asking for another language shows you a public page as the site serves it to readers who prefer that language. It is not a way to reach content behind a login, a paywall or another access control: logins, paywalls and private APIs are out of scope. Check robots.txt and the site's terms of service before you read a site at volume. This is general information, not legal advice.

Where Live Web Data API fits

In Live Web Data API, countryCode sets the country the page is fetched from, and the default is us. Every request fetches the page live; there is no cache. Record the country with the language signals above, so a page read from one country is never mistaken for another. The API reference describes the parameter.

Frequently asked questions

What happens if a fetcher sends no Accept-Language header?

RFC 9110 says a request without the field implies that the sender has no preference on that dimension of negotiation. The site then serves whatever version it chooses when no language is requested.

Does sending Accept-Language make the page come back in that language?

Not always. RFC 9110 lets the origin server either answer 406 (Not Acceptable) or disregard the header field. Some sites never look at it and choose a language another way, for example in the browser after the page loads.

Is Content-Language the language the page is written in?

Not necessarily. RFC 9110 defines it as the natural language of the intended audience, which might not be every language used in the page. The W3C recommends the lang attribute on the html element to declare the language of the text, and says that attribute always overrides the HTTP header.

How should I tell the model which language a page is in?

Store the language you detected next to the URL, country and fetch time, and put it in the source label of the prompt. Then ask the model to answer in the user's language and to say when the source was in another one.

Sources

  1. RFC 9110: HTTP Semantics (Documentation)
  2. Content-Language (MDN) (Documentation)
  3. HTTP headers, meta elements and language information (W3C Internationalization) (Documentation)
  4. T275597: Portal ignores Accept-Language header, content always in English language (News)