Why coding assistants suggest deprecated APIs, and what docs fix

A coding assistant suggests a call, the code fails, and the error says the function no longer exists. The model did not invent it. It learned a real API that the library has since renamed, moved or removed. Its knowledge stops at its training data, and the docs page that would correct it changed after that. This post looks at what two studies measured, what docs in context fix and what they do not, and where one live read of the current page belongs.

What the research measured

The clearest measurement is an ICSE 2025 paper, "LLMs Meet Library Evolution". The authors built 145 mappings between deprecated APIs and their replacements across 8 Python libraries (NumPy, Pandas, scikit-learn, SciPy, seaborn, TensorFlow, PyTorch and Transformers), then ran 28,125 completion prompts through 7 models.

Across all prompts, the share of completions that used the deprecated form ranged from 24.9% (CodeGen-350m) to 37.4% (GPT-3.5). That is roughly 1 deprecated call in every 3 to 4 completions where both forms were possible.

The paper names three causes:

• Training data. Deprecated calls are common in open-source code, so models learn them as normal usage.
• No deprecation signal at inference. Nothing in the prompt tells the model a call is outdated.
• Small renames. When the replacement looks almost like the old call, models mix them up more often. The paper's example is PyTorch moving torch.lstsq to torch.linalg.lstsq.

The context around the cursor matters most

The most useful number for an app builder is the split by prompt. When the prompt came from functions that already used outdated code, the deprecated usage rate was 70% to 90%. When it came from up-to-date functions, it was 9% to 18%.

The decision follows from that. The risk is highest exactly when the user is working in an old codebase, which is also when they most need the current API. On these benchmarks, trusting the model's memory of the library in that situation produced the old call 70% to 90% of the time.

What docs in context fix, and what they do not

The same paper tested a prompt-side fix, InsertPrompt, which adds the replacement API information to the prompt. Fix rates ranged from 25.7% on the smallest model to 97.2% on GPT-3.5, and the authors still judged it not accurate enough on its own.

A 2026 study, "When LLMs Lag Behind", tested this at a larger scale: 270 real API updates (deprecations, changes and additions) from 8 Python libraries, across 11 models from 4 families. Without full documentation, 42.55% of the generated examples ran in the target environment. With structured update documentation and larger models, the best result was 66.36%. Self-reflection prompting raised the executable rate by a further 11%, as the abstract reports it. The authors also report that outdated patterns persist even when the update specification is in the prompt.

So the honest summary is: the current docs in context move the model a long way, from about 4 in 10 runnable examples toward 7 in 10 in that study, but they do not replace running the code.

A rule for when to read the docs page live

Start with this default: read the current reference page live when any one of these is true, and skip it otherwise.

1. The question names a library and a function, class or argument.
2. The code around the cursor imports a library that has had a major release since the model's training cutoff.
3. The last suggestion failed with an attribute, import or signature error.

Pass the page in the same block format each time, with the version and the fetch time, so the model can tell current docs from its memory:

[docs]
url: https://example.com/docs/2.15/linalg/lstsq
library_version: 2.15
fetched_at: 2026-11-09T08:30:00Z
---
torch.linalg.lstsq(A, B, rcond=None, *, driver=None)
Returns a named tuple (solution, residuals, rank, singular_values).
---
Use the signature above. If your memory of this API differs, the docs block wins.

The signature shown is the one on the PyTorch 2.15 reference page for torch.linalg.lstsq: A and B first, rcond optional and driver keyword-only. Keep the excerpt to the section that answers the question rather than the whole page; the token cost post shows how to estimate what a full page would add.

Then run the code. The 2026 study is clear that docs raise the success rate but still leave about 1 example in 3 that does not run, so the error message is your second source of truth.

When a stored copy is enough

A docs page changes when the library releases, not every hour. A stored copy is fine while the project pins the same version. Record the version and the fetch time with the copy, and read the page again when the project upgrades the dependency or the copy is older than the max age you chose. The guide on why LLM apps need live web data covers the broader live-or-stored decision.

Where Live Web Data API fits

Live Web Data API returns a public page, such as a library's reference page, as Markdown, plain text or HTML. Every request fetches the page live; there is no cache. Each URL comes back with statuses[].fetchedAt, which fills the fetched_at line above, and text.maxCharacters caps how much of a long reference page goes into context. It reads pages you name; it does not search for the right docs page or crawl a docs site. To see what one reference page looks like as Markdown before you wire it in, try a URL in the playground.