# Large Language Models Lack Temporal Awareness of Medical Knowledge

arXiv: https://arxiv.org/abs/2605.13045 · HTML: https://mlsanity.com/paper/2605.13045/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 46 references that are in the map.
- First posted on arXiv 13 May 2026.

## Builds on (most cited first, 29 of 29)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering](https://mlsanity.com/paper/2203.14371/) (2022)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [MedGemma Technical Report](https://mlsanity.com/paper/2507.05201/) (2025)
- [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://mlsanity.com/paper/2505.08775/) (2025)
- [Kimi-VL Technical Report](https://mlsanity.com/paper/2504.07491/) (2025)
- [Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions](https://mlsanity.com/paper/2308.11483/) (2023)
- [MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding](https://mlsanity.com/paper/2501.18362/) (2025)
- [MedExQA: Medical Question Answering Benchmark with Multiple Explanations](https://mlsanity.com/paper/2406.06331/) (2024)
- [Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models](https://mlsanity.com/paper/2407.16221/) (2024)
- [Biomedical Question Answering: A Survey of Approaches and Challenges](https://mlsanity.com/paper/2102.05281/) (2021)
- [MedGemma 1.5 Technical Report](https://mlsanity.com/paper/2604.05081/) (2026)
- [Large Language Models-guided Dynamic Adaptation for Temporal Knowledge Graph Reasoning](https://mlsanity.com/paper/2405.14170/) (2024)
- [Towards Evaluating and Building Versatile Large Language Models for Medicine](https://mlsanity.com/paper/2408.12547/) (2024)
- [TIMER: Temporal Instruction Modeling and Evaluation for Longitudinal Clinical Records](https://mlsanity.com/paper/2503.04176/) (2025)
- [The Distracting Effect: Understanding Irrelevant Passages in RAG](https://mlsanity.com/paper/2505.06914/) (2025)
- [DyKnow: Dynamically Verifying Time-Sensitive Factual Knowledge in LLMs](https://mlsanity.com/paper/2404.08700/) (2024)
- [ToolUniverse: An open platform for democratizing AI scientists](https://mlsanity.com/paper/2509.23426/) (2025)
- [Temporal Knowledge Question Answering via Abstract Reasoning Induction](https://mlsanity.com/paper/2311.09149/) (2023)
- [EvoWiki: Evaluating LLMs on Evolving Knowledge](https://mlsanity.com/paper/2412.13582/) (2024)
- [Assessing and Mitigating Medical Knowledge Drift and Conflicts in Large Language Models](https://mlsanity.com/paper/2505.07968/) (2025)
- [Temporal Reasoning with Large Language Models Augmented by Evolving Knowledge Graphs](https://mlsanity.com/paper/2509.15464/) (2025)
- [When Facts Change: Probing LLMs on Evolving Knowledge with evolveQA](https://mlsanity.com/paper/2510.19172/) (2025)

## Built on by (most cited first, 2 of 2)

- [CLEAR: Cross-Source Evidence Adjudication for Large Language Models in Medicine](https://mlsanity.com/paper/2609.16301/) (2026)
- [Stale-Document Poisoning: When Outdated Retrieval Overrides Correct Model Answers](https://mlsanity.com/paper/2609.31342/) (2026)

## Abstract (arXiv)

The existing methods for evaluating the medical knowledge of Large Language Models (LLMs) are largely based on atemporal examination-style benchmarks, while in reality, medical knowledge is inherently dynamic and continuously evolves as new evidence emerges and treatments are approved. Consequently, evaluating medical knowledge without a temporal context may provide an incomplete assessment of whether LLMs can accurately reason about time-specific medical knowledge. Moreover, most medical data are historical, requiring the models not only to recall the correct knowledge, but also to know when that knowledge is correct. To bridge the gap, we built TempoMed-Bench, the first-of-its-kind benchmark for evaluating the temporal awareness of the LLMs in the medical domain through evolving guideline knowledge. Based on the TempoMed-Bench, our evaluation analysis first reveals that LLMs lack temporal awareness in medical knowledge through the key findings: (1) model performance on up-to-date medical knowledge exhibits a gradual linear decline over time rather than a sharp knowledge-cutoff behavior, suggesting that parametric medical knowledge is not strictly bounded by knowledge cutoffs; (2) LLMs consistently struggle more with recalling outdated historical medical knowledge than with up-to-date recommendations: accuracy of historical knowledge is only 25.37%-53.89% of up-to-date knowledge, indicating potential knowledge forgetting effects during training; and (3) LLMs often exhibit temporally inconsistent behaviors, where predictions fluctuate irregularly across neighboring years. We also show that the temporal awareness problem is a challenge that cannot be easily solved when integrated with agentic search tools (-3.15%-14.14%). This work highlights an important yet underexplored challenge and motivates future research on developing LLMs that can better encode time-specific medical knowledge.