Pages by country
Locale-adaptive pages: what a fetcher from one country never sees
Learn why a page that adapts to country or language shows a one-country fetcher only one version, how to spot such pages and which version to pass to a model.
By Curtis Mccann, Live Web Data team · Published
This article was drafted with AI assistance from the sources listed below and edited and published by the Live Web Data team. Disclosure: the Live Web Data API, which this article mentions, is this site's own product.
- Read the same URL from a second country and compare the text
- Look for hreflang alternates that point to separate locale URLs
- Compare the final URL with the URL you asked for
- Fetch the locale URL directly when the site has one
- Store and pass the country the page was read from
A locale-adaptive page changes what it shows at the same URL, based on where the visitor seems to be or which language they prefer. A reader in the US and a reader in Germany open one address and get two pages. A fetcher that always reads from one country gets one of them, every time, and has no reason to suspect the other exists. The model then answers from that one version for every user.
Google documents this exact blind spot for its own crawler. The same reasoning applies to any app that reads the web from one place.
What Google says its crawler misses
Google's Search Central page on locale-adaptive pages says Google may not crawl, index or rank all of the content for every locale on such a site. It gives two reasons. First: "the default IP addresses of the Googlebot crawler appear to be based in the USA." Second: "the crawler sends HTTP requests without setting Accept-Language in the request header."
So a site that adapts by country shows Googlebot the US version by default, and a site that adapts by language shows it whatever the site serves when no language is requested. Google adds that Googlebot also crawls from IP addresses based outside the USA, with the same user agent string for all crawling configurations. Its main recommendation to site owners is still to give each locale its own URL and annotate those URLs with hreflang.
The same blind spot in your fetcher
Your fetcher has the same two inputs: the country it connects from and the headers it sends. If both are fixed, it sees one variant of every adaptive page. The address in the result can be the same for every variant, so the URL alone does not tell you which one you got.
That is fine when your users are in the same market as your fetcher. It goes wrong when a user in another country asks about a price, a plan or a return policy, and the answer comes from a page meant for someone else. The model is not at fault here: it answered from the page it was given. The guide on web pages by country lists what typically changes between versions.
How sites decide which version to serve
Sites adapt in two common ways:
- By the connection's location. The server or its CDN looks at where the request comes from.
- By language preference. The browser sends an
Accept-Languageheader, and the site picks a language from it. A request without that header gets the site's fallback choice.
Google's guide to multi-regional sites advises site owners against both habits. It says "Don't use IP analysis to adapt your content", recommends different URLs for each language version rather than cookies or browser settings, and says "don't redirect based on what you think the user's language may be." This is advice to site owners. It tells you what Google recommends, not how every site behaves, so check the pages you depend on.
Check whether a page is locale-adaptive
Three checks cover most cases:
- Read the URL from two countries and compare the text. Look at language, currency, tax display, legal notices and consent text. Markdown output makes the difference easy to diff.
- Look for hreflang alternates. If the page's HTML lists alternate URLs per language or region, the site has separate locale URLs. Fetch the one you need instead of the adaptive one.
- Compare the final URL with the URL you asked for. A redirect to a path such as
/de/means the site picked a version for you. The post on final URL vs canonical URL covers which address to keep.
If the text is the same from both countries, there are no alternates and no redirect, treat the page as one version for now and record the country you read it from anyway.
Which version to pass to the model
Once you know a page adapts, decide on purpose:
- Prefer a locale URL. When the site has a separate URL for the locale you answer for, fetch that URL. Its content does not depend on where your fetcher sits.
- Otherwise read from the country you answer for. That is usually the user's country, or the one the question names.
- Store copies by URL and country. Two reads of one URL from two countries are two different documents.
- Pass the country with the content. Put it next to the URL and fetch time, and tell the model to answer for that market only.
<source url="https://example.com/pricing" country="de" fetched="2026-11-16T08:30:00Z">
...page text...
</source>
Answer for readers in the country shown on the source. If the question is about another country, say that this page was read from a different one.
- Do not treat the US read as the real page. For an adaptive site, every locale's version is equally real. The US one is only the default for many fetchers.
Public pages and the site's rules
Reading a page from another country shows you the public page as readers there see it. It is not a way to reach content a site restricts by login, paywall or access control: logins, paywalls and private APIs are out of scope. Google's page also asks site owners to apply robots rules consistently, so that "robots meta tags and the robots.txt file must specify the same rules in each locale." On the reading side, check robots.txt and the site's terms of service for every country you read from. This is general information, not legal advice.
Where Live Web Data API fits
In Live Web Data API, countryCode sets the country the page is fetched from, and the default is us. Every request fetches the page live; there is no cache. The API reference describes the parameter.
Frequently asked questions
What is a locale-adaptive page?
A page that returns different content at the same URL depending on the country the visitor seems to come from or the language they prefer. Google's documentation uses this term for such pages.
Why does Google say it may miss content on locale-adaptive pages?
Google's documentation says the default IP addresses of Googlebot appear to be based in the USA and that the crawler sends requests without an Accept-Language header. A site that adapts by country or language may therefore show it only one version, so Google may not crawl, index or rank every locale's content.
How can I tell if a page adapts to the reader's country?
Read the same URL from two countries and compare the text: language, currency and legal notices. Also check the HTML for hreflang alternates that point to separate locale URLs, and compare the final URL after redirects with the one you asked for.
Which version should my AI app pass to the model?
The version for the country you answer for. If the site has a separate URL for that locale, fetch that URL. Otherwise read the adaptive URL from that country, and pass the country to the model next to the content.