mlsanityData to 14 Oct 2026

Facts Fade Fast: Evaluating Memorization of Outdated Medical Knowledge in Large Language Models

arXiv 2509.04304 ↗First posted 4 Sep 2025cs.CL cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

The growing capabilities of Large Language Models (LLMs) show significant potential to enhance healthcare by assisting medical researchers and physicians. However, their reliance on static training data is a major risk when medical recommendations evolve with new research and developments. When LLMs memorize outdated medical knowledge, they can provide harmful advice or fail at clinical reasoning tasks. To investigate this problem, we introduce two novel question-answering (QA) datasets derived from systematic reviews: MedRevQA (16,501 QA pairs covering general biomedical knowledge) and MedChangeQA (a subset of 512 QA pairs where medical consensus has changed over time). Our evaluation of eight prominent LLMs on the datasets reveals consistent reliance on outdated knowledge across all models. We additionally analyze the influence of obsolete pre-training data and training strategies to explain this phenomenon and propose future directions for mitigation, laying the groundwork for developing more current and reliable medical AI systems.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 32 of its 56 references that are in the map. First posted on arXiv 4 Sep 2025.

Builds on

32 of 56 references.

References · 32

  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  2. 2023A Comprehensive Overview of Large Language Modelscited by 648
  3. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  4. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  5. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 325
  6. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 318
  7. 2023Evaluating Verifiability in Generative Search Enginescited by 303
  8. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 301
  9. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 296
  10. 2024Knowledge Conflicts for LLMs: A Surveycited by 283
  11. 2022RealTime QA: What's the Answer Right Now?cited by 244
  12. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
20 more
  1. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 202
  2. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 153
  3. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 151
  4. 2022Measuring Forgetting of Memorized Training Examplescited by 118
  5. 2024LLM Dataset Inference: Did you train on my dataset?cited by 110
  6. 2023PMC-LLaMA: Towards Building Open-source Language Models for Medicinecited by 88
  7. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  8. 2023LatestEval: Addressing Data Contamination in Language Model Evaluation through Dynamic and Time-Sensitive Test Constructioncited by 58
  9. 2024On Large Language Model Continual Unlearningcited by 52
  10. 2022Annotation Error Detection: Analyzing the Past and Present for a More Coherent Futurecited by 45
  11. 2024Learning to Edit: Aligning LLMs with Knowledge Editingcited by 43
  12. 2021AmbiFC: Fact-Checking Ambiguous Claims with Evidencecited by 38
  13. 2023HealthFC: Verifying Health Claims with Evidence-Based Medical Fact-Checkingcited by 35
  14. 2024Is Your LLM Outdated? A Deep Look at Temporal Generalizationcited by 26
  15. 2024Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarkscited by 17
  16. 2024Missci: Reconstructing Fallacies in Misrepresented Sciencecited by 14
  17. 2024MedREQAL: Examining Medical Knowledge Recall of Large Language Models via Question Answeringcited by 9
  18. 2024Understanding Fine-grained Distortions in Reports of Scientific Findingscited by 8
  19. 2024ChroKnowledge: Unveiling Chronological Knowledge of Language Models in Multiple Domainscited by 8
  20. 2024M-QALM: A Benchmark to Assess Clinical Reading Comprehension and Knowledge Recall in Large Language Models via Question Answeringcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Cross-Lingual Empirical Evaluation of Large Language Models for Arabic Medical Taskscited by 2
  2. 2026Asking For An Old Friend: Diagnosing and Mitigating Temporal Failure Modes in LLM-based Statutory Question Answeringcited by 1
  3. 2026Does Finetuning with Scientific Data Increase Hallucinations? A Multi-domain Factuality Evaluation of LLMscited by 0
  4. 2026Epi-Logic: A Conceptual Framework for Epistemic Runtime Control, Schema Validity Checking, and Controlled Accommodation in Autonomous AI Agentscited by 0