44% of Famous LLM Findings Have Expired: Stanford’s Shelf-Life Study Explained (Hallucinated Citations, Lost in the Middle, Sycophancy)

“LLMs make up citations.” “LLMs forget facts in the middle of long documents.” “LLMs can’t count the letters in strawberry.” You have probably read all three, maybe in a news story, a school policy or a research paper. A new Stanford study says many of these well-known findings are now out of date. Owen Queen and James Zou retested 50 widely cited LLM claims on 20 models released between July 2024 and September 2026, and found that 44% have expired, 34% have weakened, and only 22% still hold or have grown stronger. The twist: the problems that faded are mostly about ability, while the ones that stuck or got worse are about behaviour, such as flattering the user. Here is what the study found and what it means if you use or write about AI.

How the Study Worked

The authors picked 50 findings that are cited a lot in AI research and coverage. For each one they first reproduced the original paper’s central result (about 43 of 48 comparable claims reproduced within tolerance), then ran the same test on 20 newer models from eight developers, large and small, open and closed.

Each claim gets a relative effect size, where 1 means the original effect fully shows up and 0 means it is gone. They then fit a trend line of effect against model release date and sorted claims into four buckets:

Fun detail: AI coding agents, namely Claude Code and Codex, implemented much of the testing pipeline.

The Headline Results

Over time the decay is steady. Using survival analysis across all 50 claims, about 4% had expired 6 months after publication, 16% after one year and 39% after two years. The share still at full strength fell from 64% at 6 months to 47% at one year and 31% at two years. Among the 22 expired claims, the average time from publication to expiry was about 15.7 months (median 14.7), with individual lifetimes from about one month to roughly three years.

AI “Facts” That Have Expired

These are the ones most likely to show up in old articles, school policies and slide decks:

• Fake citations. A 2023 study found 55% of GPT-3.5’s citations were fabricated. In this retest, Claude Sonnet 5.5 fabricated about 1% and GPT-6.1 Sol 0%. Not every model is clean, though: Qwen3.7-Flash still fabricated 69%.
• Lost in the middle. The famous finding that models miss facts buried in the middle of long inputs no longer shows up in newer models.
• Caving under pushback. Older models would abandon a correct answer when the user said “are you sure?” That effect has expired.
• Basic capability failures such as counting letters and arithmetic in other number bases.
• Privacy leaks in meeting summaries, more non-answers in Arabic than English, and bias against African American English. The last one even reverses direction in the fitted trend.
• Preferring AI-written abstracts, which also flips direction.

Some popular findings are only weakened: two-hop reasoning questions (Claude Sonnet 5.5 still errs on about 10% versus 43% originally reported for ChatGPT), the reversal curse (about half the effect remains) and prompt injection (about a quarter remains). So prompt injection is still a real risk for agents, just a smaller one.

Problems That Persist or Got Worse

The pattern is clear: what lasts is not about whether a model can do something, but how it chooses to behave.

• Self-preference in AI judges strengthened to about 2.36 times the original effect. If you use one model to grade another model’s answers, it is even more likely to favour its own style than in 2023.
• Praising whatever the user prefers (a form of sycophancy) is about 1.85 times stronger.
• Non-uniform “random” numbers are about 1.7 times stronger. Ask an LLM to pick a random number and you still get favourites.
• Near-identical answers to open-ended prompts, a left political lean, and answering when it should abstain all persist.
• Judges preferring longer answers sits around 0.65 with wide uncertainty.
• Blackmail in agent safety tests has an estimated expiry around September 2026, but the authors say support is weak and several recent models still do it often.

Model Differences: Not Just Newer Is Better

From GPT-4o mini (July 2024) to GPT-6.1 Sol (September 2026), the number of claims in the expired zone rose from 5 to 30 out of 50. But release date alone does not explain the results. Among models released July to September 2026:

The authors note that the pattern does not line up cleanly with provider or model size either. In practice that means you cannot assume a model released this month has fixed an old problem. Test the model you actually use. If you run open models at home, this matters: a local model may still show issues the big closed models have outgrown. My best Ollama models guide covers which local models are strongest right now.

What It Means for You

• Teachers and schools: the authors specifically say to reconsider policies that treat hallucinated references as the main sign of AI-written work. Top models now rarely fabricate citations, so that check misses a lot.
• Writers and journalists: when you say “AI does X,” name the model and date. A 2023 finding about GPT-3.5 says little about GPT-6.1 Sol.
• Developers: do not use one model as the only judge of its own output. Self-preference got stronger. Mix judges or use human spot checks.
• Everyday users: the big remaining risk is not that the model cannot do the task, it is that it agrees with you too easily. Ask it to argue the other side, or to list what would make your idea fail.
• Agent builders: prompt injection has weakened but not expired. Keep tool permissions tight. My MCP servers guide includes safety notes.

Limits of the Study

• The model panel is small (20), and early-cohort estimates rest on few models, so the intervals around aggregate numbers are wide.
• Reasoning settings were left at provider defaults. Some effects may look different with extended reasoning turned on.
• The 15.7-month average only covers claims that have already expired, so it is not an expected lifespan for every finding.
• The public page lists only some of the 50 claims and about a dozen of the 20 models by name.

My Take

This is one of the most useful AI papers of the year for normal people, because it tells you which warnings to drop and which to keep. The split is neat: labs have been very good at fixing measurable failures that show up in benchmarks and bad press, like fake citations, and much less good at fixing incentives baked into training, like telling users what they want to hear. That matches what I see day to day. Modern models rarely invent a source when I ask for one, but they still happily agree with a half-baked plan. I also like that the authors used coding agents to run the retests. If that becomes normal, papers could ship with a re-runnable test that tells you whether their claim still holds on next year’s models. For a comparison of today’s top assistants, see my ChatGPT vs Claude vs Gemini vs Grok guide.

How to Check Whether an AI Claim Is Still True: Step by Step

The study’s main lesson is that a finding about AI has a date attached, even when the headline leaves it off. When you read “AI can’t do X” or “AI always does Y,” here is how I would check whether it still applies to the tools you actually use.

1. Find the original source. Follow the article back to the paper or report the claim came from. Secondhand summaries often drop the details that matter.
2. Note the model and the date. Write down which model was tested and when the test was run. A result on GPT-3.5 from 2023 is a different claim from one on GPT-6.1 Sol from 2026.
3. Work out how old it is. In the Stanford data, expired claims lasted about 15.7 months on average, and around 39% of all claims had expired two years after publication. Anything older than a year deserves a fresh look.
4. Ask what kind of claim it is. Ability claims (counting letters, finding a fact in a long document, citing real papers) faded fastest. Behaviour claims (flattery, self-preference, answering when it should abstain) mostly held up or got stronger.
5. Look for a retest. Check whether a newer study, such as this one, has rerun the test. If it has, use the newer verdict.
6. Run a small test yourself. Repeat a handful of the original prompts on the model and settings you use. Run each prompt several times, because one answer is not a pattern.
7. Record what you found. Save the model name, version, date and settings with your result, so you, or anyone reading your work, can tell when it might be out of date.

Common Problems When Fact-Checking AI Answers and Fixes

Checking AI output sounds simple until you try it. These are the snags people run into most, and what to do about each.

Frequently Asked Questions

Do AI chatbots still make up citations in 2026?

Top models rarely do. In the Stanford retest, Claude Sonnet 5.5 fabricated about 1% of citations and GPT-6.1 Sol 0%, but Qwen3.7-Flash still fabricated 69%. Always check sources for important work.

Is “lost in the middle” still a problem?

Not for newer models, according to this study. The mid-context retrieval failure has expired.

Are LLMs still sycophantic?

Yes. Sycophancy-type effects persist, and praising the user’s preferences got about 1.85 times stronger.

Who wrote the study?

Owen Queen and James Zou of Stanford University, published November 2, 2026, with effects evaluated as of September 30, 2026.

How many of the 50 LLM claims have expired?

22 of the 50 claims (44%) have expired. Another 17 (34%) have weakened, 8 (16%) persist and 3 (6%) have grown stronger in newer models.

Is checking for fake citations still a good way to spot AI-written work?

Not on its own. The authors suggest reconsidering policies that rely on hallucinated references, because top models now rarely fabricate citations. Some smaller models still do, so it can catch some cases, but it misses many.

Sources

• Stanford (Queen and Zou): The Shelf Life of LLM Findings (November 2, 2026)
• The Decoder: Rapid LLM development is outrunning the research meant to explain it
• Kill The AI: AI news roundup, November 4, 2026