# MuLan: A Study of Fact Mutability in Language Models

arXiv: https://arxiv.org/abs/2404.03036 · HTML: https://mlsanity.com/paper/2404.03036/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 26 references that are in the map.
- First posted on arXiv 3 Apr 2024.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [Time-Aware Language Models as Temporal Knowledge Bases](https://mlsanity.com/paper/2106.15110/) (2021)
- [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](https://mlsanity.com/paper/2104.08315/) (2021)
- [Evaluating the Ripple Effects of Knowledge Editing in Language Models](https://mlsanity.com/paper/2307.12976/) (2023)
- [Information-Theoretic Probing with Minimum Description Length](https://mlsanity.com/paper/2003.12298/) (2020)
- [X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models](https://mlsanity.com/paper/2010.06189/) (2020)
- [LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development](https://mlsanity.com/paper/2305.07507/) (2023)
- [Are Large Language Models Temporally Grounded?](https://mlsanity.com/paper/2311.08398/) (2023)

## Built on by (most cited first, 10 of 10)

- [DYNAMICQA: Tracing Internal Knowledge Conflicts in Language Models](https://mlsanity.com/paper/2407.17023/) (2024)
- [Fundamental Problems With Model Editing: How Should Rational Belief Revision Work in LLMs?](https://mlsanity.com/paper/2406.19354/) (2024)
- [Unified Active Retrieval for Retrieval Augmented Generation](https://mlsanity.com/paper/2406.12534/) (2024)
- [Defining Knowledge: Bridging Epistemology and Large Language Models](https://mlsanity.com/paper/2410.02499/) (2024)
- [Will It Still Be True Tomorrow? Multilingual Evergreen Question Classification to Improve Trustworthy QA](https://mlsanity.com/paper/2505.21115/) (2025)
- [PASS-FC: Progressive and Adaptive Search Scheme for Fact Checking of Comprehensive Claims](https://mlsanity.com/paper/2504.09866/) (2025)
- [From Word to World: Evaluate and Mitigate Culture Bias in LLMs via Word Association Test](https://mlsanity.com/paper/2505.18562/) (2025)
- [Temporal Fact Conflicts in LLMs: Reproducibility Insights from Unifying DYNAMICQA and MULAN](https://mlsanity.com/paper/2603.15892/) (2026)
- [Right Knowledge, Wrong Answer: Characterizing Parametric Temporal Conflict in Open-Weight Language Models](https://mlsanity.com/paper/2606.20959/) (2026)
- [Do We Really Need GNNs with Explicit Structural Modeling? MLPs Suffice for Language Model Representations](https://mlsanity.com/paper/2506.21682/) (2025)

## Abstract (arXiv)

Facts are subject to contingencies and can be true or false in different circumstances. One such contingency is time, wherein some facts mutate over a given period, e.g., the president of a country or the winner of a championship. Trustworthy language models ideally identify mutable facts as such and process them accordingly. We create MuLan, a benchmark for evaluating the ability of English language models to anticipate time-contingency, covering both 1:1 and 1:N relations. We hypothesize that mutable facts are encoded differently than immutable ones, hence being easier to update. In a detailed evaluation of six popular large language models, we consistently find differences in the LLMs' confidence, representations, and update behavior, depending on the mutability of a fact. Our findings should inform future work on the injection of and induction of time-contingent knowledge to/from LLMs.