# When Facts Change: Probing LLMs on Evolving Knowledge with evolveQA

arXiv: https://arxiv.org/abs/2510.19172 · HTML: https://mlsanity.com/paper/2510.19172/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 22 Oct 2025.

## Builds on (most cited first, 26 of 26)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts](https://mlsanity.com/paper/2305.13300/) (2023)
- [Time-Aware Language Models as Temporal Knowledge Bases](https://mlsanity.com/paper/2106.15110/) (2021)
- [Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence](https://mlsanity.com/paper/2210.13701/) (2022)
- [Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning](https://mlsanity.com/paper/2406.09170/) (2024)
- [ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLM](https://mlsanity.com/paper/2408.12076/) (2024)
- [Clio: Privacy-Preserving Insights into Real-World AI Use](https://mlsanity.com/paper/2412.13678/) (2024)
- [ClusterLLM: Large Language Models as a Guide for Text Clustering](https://mlsanity.com/paper/2305.14871/) (2023)
- [Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Method](https://mlsanity.com/paper/2310.17918/) (2023)
- [Attacking Open-domain Question Answering by Injecting Misinformation](https://mlsanity.com/paper/2110.07803/) (2021)
- [DYNAMICQA: Tracing Internal Knowledge Conflicts in Language Models](https://mlsanity.com/paper/2407.17023/) (2024)
- [Studying Large Language Model Behaviors Under Context-Memory Conflicts With Real Documents](https://mlsanity.com/paper/2404.16032/) (2024)
- [Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Models](https://mlsanity.com/paper/2311.08106/) (2023)
- [EvoWiki: Evaluating LLMs on Evolving Knowledge](https://mlsanity.com/paper/2412.13582/) (2024)
- [TimelineQA: A Benchmark for Question Answering over Timelines](https://mlsanity.com/paper/2306.01069/) (2023)
- [Talking Point based Ideological Discourse Analysis in News Events](https://mlsanity.com/paper/2504.07400/) (2025)
- [Question Answering under Temporal Conflict: Evaluating and Organizing Evolving Knowledge with LLMs](https://mlsanity.com/paper/2506.07270/) (2025)
- [Towards Corpus-Scale Discovery of Selection Biases in News Coverage: Comparing What Sources Say About Entities as a Start](https://mlsanity.com/paper/2304.03414/) (2023)
- [Is Large Language Model Performance on Reasoning Tasks Impacted by Different Ways Questions Are Asked?](https://mlsanity.com/paper/2507.15707/) (2025)

## Built on by (most cited first, 2 of 2)

- [Large Language Models Lack Temporal Awareness of Medical Knowledge](https://mlsanity.com/paper/2605.13045/) (2026)
- [Time Present and Time Past: Benchmarking Large Language Models on Temporally Evolving Document Understanding](https://mlsanity.com/paper/2608.08512/) (2026)

## Abstract (arXiv)

LLMs often fail to handle temporal knowledge conflicts--contradictions arising when facts evolve over time within their training data. Existing studies evaluate this phenomenon through benchmarks built on structured knowledge bases like Wikidata, but they focus on widely-covered, easily-memorized popular entities and lack the dynamic structure needed to fairly evaluate LLMs with different knowledge cut-off dates. We introduce evolveQA, a benchmark specifically designed to evaluate LLMs on temporally evolving knowledge, constructed from 3 real-world, time-stamped corpora: AWS updates, Azure changes, and WHO disease outbreak reports. Our framework identifies naturally occurring knowledge evolution and generates questions with gold answers tailored to different LLM knowledge cut-off dates. Through extensive evaluation of 12 open and closed-source LLMs across 3 knowledge probing formats, we demonstrate significant performance drops of up to 31% on evolveQA compared to static knowledge questions.