# EvoWiki: Evaluating LLMs on Evolving Knowledge

arXiv: https://arxiv.org/abs/2412.13582 · HTML: https://mlsanity.com/paper/2412.13582/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 40 references that are in the map.
- First posted on arXiv 18 Dec 2024.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts](https://mlsanity.com/paper/2305.13300/) (2023)
- [Generate rather than Retrieve: Large Language Models are Strong Context Generators](https://mlsanity.com/paper/2209.10063/) (2022)
- [Time-Aware Language Models as Temporal Knowledge Bases](https://mlsanity.com/paper/2106.15110/) (2021)
- [RealTime QA: What's the Answer Right Now?](https://mlsanity.com/paper/2207.13332/) (2022)
- [When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method](https://mlsanity.com/paper/2402.17193/) (2024)
- [Towards Continual Knowledge Learning of Language Models](https://mlsanity.com/paper/2110.03215/) (2021)
- [A Dataset for Answering Time-Sensitive Questions](https://mlsanity.com/paper/2108.06314/) (2021)
- [TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models](https://mlsanity.com/paper/2204.14211/) (2022)
- [Can Large Language Models Really Improve by Self-critiquing Their Own Plans?](https://mlsanity.com/paper/2310.08118/) (2023)
- [Lift Yourself Up: Retrieval-augmented Text Generation with Self Memory](https://mlsanity.com/paper/2305.02437/) (2023)
- [Question Answering Over Temporal Knowledge Graphs](https://mlsanity.com/paper/2106.01515/) (2021)
- [SCATTER: Selective Context Attentional Scene Text Recognizer](https://mlsanity.com/paper/2003.11288/) (2020)
- [Complex Temporal Question Answering on Knowledge Graphs](https://mlsanity.com/paper/2109.08935/) (2021)
- [Instruction-tuned Language Models are Better Knowledge Learners](https://mlsanity.com/paper/2402.12847/) (2024)
- [TEQUILA: Temporal Question Answering over Knowledge Bases](https://mlsanity.com/paper/1908.03650/) (2019)
- [DYNAMICQA: Tracing Internal Knowledge Conflicts in Language Models](https://mlsanity.com/paper/2407.17023/) (2024)
- [Context versus Prior Knowledge in Language Models](https://mlsanity.com/paper/2404.04633/) (2024)
- [Structure-Aware Language Model Pretraining Improves Dense Retrieval on Structured Data](https://mlsanity.com/paper/2305.19912/) (2023)
- [Intuitive or Dependent? Investigating LLMs' Behavior Style to Conflicting Prompts](https://mlsanity.com/paper/2309.17415/) (2023)
- [PAT-Questions: A Self-Updating Benchmark for Present-Anchored Temporal Question-Answering](https://mlsanity.com/paper/2402.11034/) (2024)
- [A + B: A General Generator-Reader Framework for Optimizing LLMs to Unleash Synergy Potential](https://mlsanity.com/paper/2406.03963/) (2024)

## Built on by (most cited first, 13 of 13)

- [Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks](https://mlsanity.com/paper/2504.18838/) (2025)
- [When Large Multimodal Models Confront Evolving Knowledge: Challenges and Explorations](https://mlsanity.com/paper/2505.24449/) (2025)
- [MINED: Probing and Updating with Multimodal Time-Sensitive Knowledge for Large Multimodal Models](https://mlsanity.com/paper/2510.19457/) (2025)
- [A Unified Definition of Hallucination: It's The World Model, Stupid!](https://mlsanity.com/paper/2512.21577/) (2025)
- [MMKU-Bench: A Multimodal Update Benchmark for Diverse Visual Knowledge](https://mlsanity.com/paper/2603.15117/) (2026)
- [ScienceMeter: Tracking Scientific Knowledge Updates in Language Models](https://mlsanity.com/paper/2505.24302/) (2025)
- [OpenSkillEval: Automatically Auditing the Open Skill Ecosystem for LLM Agents](https://mlsanity.com/paper/2605.23657/) (2026)
- [When Facts Change: Probing LLMs on Evolving Knowledge with evolveQA](https://mlsanity.com/paper/2510.19172/) (2025)
- [Large Language Models Lack Temporal Awareness of Medical Knowledge](https://mlsanity.com/paper/2605.13045/) (2026)
- [Time Present and Time Past: Benchmarking Large Language Models on Temporally Evolving Document Understanding](https://mlsanity.com/paper/2608.08512/) (2026)
- [Aligning Language Models with Real-time Knowledge Editing](https://mlsanity.com/paper/2508.01302/) (2025)
- [Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statements](https://mlsanity.com/paper/2512.24867/) (2025)
- [ChronoCulture: Evaluating Cultural Awareness Across Time in Large Language Models](https://mlsanity.com/paper/2610.20472/) (2026)

## Abstract (arXiv)

Knowledge utilization is a critical aspect of LLMs, and understanding how they adapt to evolving knowledge is essential for their effective deployment. However, existing benchmarks are predominantly static, failing to capture the evolving nature of LLMs and knowledge, leading to inaccuracies and vulnerabilities such as contamination. In this paper, we introduce EvoWiki, an evolving dataset designed to reflect knowledge evolution by categorizing information into stable, evolved, and uncharted states. EvoWiki is fully auto-updatable, enabling precise evaluation of continuously changing knowledge and newly released LLMs. Through experiments with Retrieval-Augmented Generation (RAG) and Contunual Learning (CL), we evaluate how effectively LLMs adapt to evolving knowledge. Our results indicate that current models often struggle with evolved knowledge, frequently providing outdated or incorrect responses. Moreover, the dataset highlights a synergistic effect between RAG and CL, demonstrating their potential to better adapt to evolving knowledge. EvoWiki provides a robust benchmark for advancing future research on the knowledge evolution capabilities of large language models.