# Fundamental Problems With Model Editing: How Should Rational Belief Revision Work in LLMs?

arXiv: https://arxiv.org/abs/2406.19354 · HTML: https://mlsanity.com/paper/2406.19354/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 64 of its 91 references that are in the map.
- First posted on arXiv 27 Jun 2024.

## Builds on (most cited first, 30 of 64)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning](https://mlsanity.com/paper/2403.03218/) (2024)
- [Fast Model Editing at Scale](https://mlsanity.com/paper/2110.11309/) (2021)
- [Memory-Based Model Editing at Scale](https://mlsanity.com/paper/2206.06520/) (2022)
- [Zero-Shot Relation Extraction via Reading Comprehension](https://mlsanity.com/paper/1706.04115/) (2017)
- [Editing Large Language Models: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2305.13172/) (2023)
- [KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](https://mlsanity.com/paper/1911.06136/) (2019)
- [Entity-Based Knowledge Conflicts in Question Answering](https://mlsanity.com/paper/2109.05052/) (2021)
- [Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts](https://mlsanity.com/paper/2305.13300/) (2023)
- [MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions](https://mlsanity.com/paper/2305.14795/) (2023)
- [Time-Aware Language Models as Temporal Knowledge Bases](https://mlsanity.com/paper/2106.15110/) (2021)
- [Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors](https://mlsanity.com/paper/2211.11031/) (2022)
- [Evaluating the Ripple Effects of Knowledge Editing in Language Models](https://mlsanity.com/paper/2307.12976/) (2023)
- [Mind the Gap: Assessing Temporal Generalization in Neural Language Models](https://mlsanity.com/paper/2102.01951/) (2021)
- [Modifying Memories in Transformer Models](https://mlsanity.com/paper/2012.00363/) (2020)
- [Knowledge Editing for Large Language Models: A Survey](https://mlsanity.com/paper/2310.16218/) (2023)
- [Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks](https://mlsanity.com/paper/2309.17410/) (2023)

## Built on by (most cited first, 24 of 24)

- [AdaCAD: Adaptively Decoding to Balance Conflicts between Contextual and Parametric Knowledge](https://mlsanity.com/paper/2409.07394/) (2024)
- [Can Knowledge Editing Really Correct Hallucinations?](https://mlsanity.com/paper/2410.16251/) (2024)
- [Beyond Prompt Engineering: Robust Behavior Control in LLMs via Steering Target Atoms](https://mlsanity.com/paper/2505.20322/) (2025)
- [A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarios](https://mlsanity.com/paper/2412.03920/) (2024)
- [How Do LLMs Acquire New Knowledge? A Knowledge Circuits Perspective on Continual Pre-Training](https://mlsanity.com/paper/2502.11196/) (2025)
- [Can Editing LLMs Inject Harm?](https://mlsanity.com/paper/2407.20224/) (2024)
- [Representation Shattering in Transformers: A Synthetic Study with Knowledge Editing](https://mlsanity.com/paper/2410.17194/) (2024)
- [OneEdit: A Neural-Symbolic Collaboratively Knowledge Editing System](https://mlsanity.com/paper/2409.07497/) (2024)
- [BeliefShift: Benchmarking Temporal Belief Consistency and Opinion Drift in LLM Agents](https://mlsanity.com/paper/2603.23848/) (2026)
- [Societal Alignment Frameworks Can Improve LLM Alignment](https://mlsanity.com/paper/2503.00069/) (2025)
- [UPCORE: Utility-Preserving Coreset Selection for Balanced Unlearning](https://mlsanity.com/paper/2502.15082/) (2025)
- [COMPKE: Complex Question Answering under Knowledge Editing](https://mlsanity.com/paper/2506.00829/) (2025)
- [RankAlign: A Ranking View of the Generator-Validator Gap in Large Language Models](https://mlsanity.com/paper/2504.11381/) (2025)
- [ThinkEval: Practical Evaluation of Knowledge Leakage in LLM Editing using Thought-based Knowledge Graphs](https://mlsanity.com/paper/2506.01386/) (2025)
- [CKnowEdit: A New Chinese Knowledge Editing Dataset for Linguistics, Facts, and Logic Error Correction in LLMs](https://mlsanity.com/paper/2409.05806/) (2024)
- [PropMEND: Hypernetworks for Knowledge Propagation in LLMs](https://mlsanity.com/paper/2506.08920/) (2025)
- [MIAT: Maneuver-Intention-Aware Transformer for Spatio-Temporal Trajectory Prediction](https://mlsanity.com/paper/2504.05059/) (2025)
- [Bilinear representation mitigates reversal curse and enables consistent model editing](https://mlsanity.com/paper/2509.21993/) (2025)
- [Dynamics Reveals Structure: Challenging the Linear Propagation Assumption](https://mlsanity.com/paper/2601.21601/) (2026)
- [Context-Fidelity Boosting: Enhancing Faithful Generation through Watermark-Inspired Decoding](https://mlsanity.com/paper/2604.22335/) (2026)
- [PASTA: A Paraphrasing And Self-Training Approach for Knowledge Updating in LLMs](https://mlsanity.com/paper/2606.28898/) (2026)
- [Tutorial on Reasoning for IR & IR for Reasoning](https://mlsanity.com/paper/2602.03640/) (2026)
- [Truth for Believable AI: Expressed Doubt, Provenance, and Belief Revision as an Engineerable Stance](https://mlsanity.com/paper/2609.26035/) (2026)
- [Stress-Testing LLM Lie Detectors: Role-Play Failures and Spurious Correlations](https://mlsanity.com/paper/2609.39807/) (2026)

## Abstract (arXiv)

The model editing problem concerns how language models should learn new facts about the world over time. While empirical research on model editing has drawn widespread attention, the conceptual foundations of model editing remain shaky -- perhaps unsurprisingly, since model editing is essentially belief revision, a storied problem in philosophy that has eluded succinct solutions for decades. Model editing nonetheless demands a solution, since we need to be able to control the knowledge within language models. With this goal in mind, this paper critiques the standard formulation of the model editing problem and proposes a formal testbed for model editing research. We first describe 12 open problems with model editing, based on challenges with (1) defining the problem, (2) developing benchmarks, and (3) assuming LLMs have editable beliefs in the first place. Many of these challenges are extremely difficult to address, e.g. determining far-reaching consequences of edits, labeling probabilistic entailments between facts, and updating beliefs of agent simulators. Next, we introduce a semi-synthetic dataset for model editing based on Wikidata, where we can evaluate edits against labels given by an idealized Bayesian agent. This enables us to say exactly how belief revision in language models falls short of a desirable epistemic standard. We encourage further research exploring settings where such a gold standard can be compared against. Our code is publicly available at: https://github.com/peterbhase/LLM-belief-revision