# WikiFactDiff: A Large, Realistic, and Temporally Adaptable Dataset for Atomic Factual Knowledge Update in Causal Language Models

arXiv: https://arxiv.org/abs/2403.14364 · HTML: https://mlsanity.com/paper/2403.14364/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 19 references that are in the map.
- First posted on arXiv 21 Mar 2024.

## Builds on (most cited first, 11 of 11)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [Large Language Models Struggle to Learn Long-Tail Knowledge](https://mlsanity.com/paper/2211.08411/) (2022)
- [Zero-Shot Relation Extraction via Reading Comprehension](https://mlsanity.com/paper/1706.04115/) (2017)
- [Editing Large Language Models: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2305.13172/) (2023)
- [Mind the Gap: Assessing Temporal Generalization in Neural Language Models](https://mlsanity.com/paper/2102.01951/) (2021)
- [Editable Neural Networks](https://mlsanity.com/paper/2004.00345/) (2020)

## Built on by (most cited first, 9 of 9)

- [WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMs](https://mlsanity.com/paper/2503.05683/) (2025)
- [BMIKE-53: Investigating Cross-Lingual Knowledge Editing with In-Context Learning](https://mlsanity.com/paper/2406.17764/) (2024)
- [ChroKnowledge: Unveiling Chronological Knowledge of Language Models in Multiple Domains](https://mlsanity.com/paper/2410.09870/) (2024)
- [MMKU-Bench: A Multimodal Update Benchmark for Diverse Visual Knowledge](https://mlsanity.com/paper/2603.15117/) (2026)
- [Better Call SAUL: Fluent and Consistent Language Model Editing with Generation Regularization](https://mlsanity.com/paper/2410.02433/) (2024)
- [Factual Knowledge in Language Models: Robustness and Anomalies under Simple Temporal Context Variations](https://mlsanity.com/paper/2502.01220/) (2025)
- [ScEdit: Script-based Assessment of Knowledge Editing](https://mlsanity.com/paper/2505.23291/) (2025)
- [RippleBench: Capturing Ripple Effects Using Existing Knowledge Repositories](https://mlsanity.com/paper/2512.04144/) (2025)
- [SCAN: Sparse Circuit Anchor Interpretable Neuron for Lifelong Knowledge Editing](https://mlsanity.com/paper/2603.15226/) (2026)

## Abstract (arXiv)

The factuality of large language model (LLMs) tends to decay over time since events posterior to their training are "unknown" to them. One way to keep models up-to-date could be factual update: the task of inserting, replacing, or removing certain simple (atomic) facts within the model. To study this task, we present WikiFactDiff, a dataset that describes the evolution of factual knowledge between two dates as a collection of simple facts divided into three categories: new, obsolete, and static. We describe several update scenarios arising from various combinations of these three types of basic update. The facts are represented by subject-relation-object triples; indeed, WikiFactDiff was constructed by comparing the state of the Wikidata knowledge base at 4 January 2021 and 27 February 2023. Those fact are accompanied by verbalization templates and cloze tests that enable running update algorithms and their evaluation metrics. Contrary to other datasets, such as zsRE and CounterFact, WikiFactDiff constitutes a realistic update setting that involves various update scenarios, including replacements, archival, and new entity insertions. We also present an evaluation of existing update algorithms on WikiFactDiff.