When Facts Change: Probing LLMs on Evolving Knowledge with evolveQA
Not on the map. Why?
Abstract (arXiv)
LLMs often fail to handle temporal knowledge conflicts--contradictions arising when facts evolve over time within their training data. Existing studies evaluate this phenomenon through benchmarks built on structured knowledge bases like Wikidata, but they focus on widely-covered, easily-memorized popular entities and lack the dynamic structure needed to fairly evaluate LLMs with different knowledge cut-off dates. We introduce evolveQA, a benchmark specifically designed to evaluate LLMs on temporally evolving knowledge, constructed from 3 real-world, time-stamped corpora: AWS updates, Azure changes, and WHO disease outbreak reports. Our framework identifies naturally occurring knowledge evolution and generates questions with gold answers tailored to different LLM knowledge cut-off dates. Through extensive evaluation of 12 open and closed-source LLMs across 3 knowledge probing formats, we demonstrate significant performance drops of up to 31% on evolveQA compared to static knowledge questions.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 26 of its 40 references that are in the map. First posted on arXiv 22 Oct 2025.
Builds on
26 of 40 references.
References · 26
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
- 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
- 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
- 2023LIMA: Less Is More for Alignmentcited by 1,080
- 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
- 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
- 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
- 2022Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidencecited by 142
- 2024Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoningcited by 83
14 more
- 2024ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLMcited by 73
- 2023ClusterLLM: Large Language Models as a Guide for Text Clusteringcited by 69
- 2024Clio: Privacy-Preserving Insights into Real-World AI Usecited by 66
- 2023Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Methodcited by 51
- 2021Attacking Open-domain Question Answering by Injecting Misinformationcited by 41
- 2024DYNAMICQA: Tracing Internal Knowledge Conflicts in Language Modelscited by 34
- 2024Studying Large Language Model Behaviors Under Context-Memory Conflicts With Real Documentscited by 27
- 2023Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Modelscited by 17
- 2024EvoWiki: Evaluating LLMs on Evolving Knowledgecited by 12
- 2023TimelineQA: A Benchmark for Question Answering over Timelinescited by 10
- 2025Talking Point based Ideological Discourse Analysis in News Eventscited by 4
- 2025Question Answering under Temporal Conflict: Evaluating and Organizing Evolving Knowledge with LLMscited by 4
- 2023Towards Corpus-Scale Discovery of Selection Biases in News Coverage: Comparing What Sources Say About Entities as a Startcited by 1
- 2025Is Large Language Model Performance on Reasoning Tasks Impacted by Different Ways Questions Are Asked?cited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.