# The Mirage of Model Editing: Revisiting Evaluation in the Wild

arXiv: https://arxiv.org/abs/2502.11177 · HTML: https://mlsanity.com/paper/2502.11177/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 48 references that are in the map.
- First posted on arXiv 16 Feb 2025.

## Builds on (most cited first, 30 of 33)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://mlsanity.com/paper/2203.14680/) (2022)
- [Zero-Shot Relation Extraction via Reading Comprehension](https://mlsanity.com/paper/1706.04115/) (2017)
- [Measuring short-form factuality in large language models](https://mlsanity.com/paper/2411.04368/) (2024)
- [Editing Large Language Models: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2305.13172/) (2023)
- [Can We Edit Factual Knowledge by In-Context Learning?](https://mlsanity.com/paper/2305.12740/) (2023)
- [Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors](https://mlsanity.com/paper/2211.11031/) (2022)
- [Modifying Memories in Transformer Models](https://mlsanity.com/paper/2012.00363/) (2020)
- [Transformer-Patcher: One Mistake worth One Neuron](https://mlsanity.com/paper/2301.09785/) (2023)
- [Knowledge Editing for Large Language Models: A Survey](https://mlsanity.com/paper/2310.16218/) (2023)
- [PMET: Precise Model Editing in a Transformer](https://mlsanity.com/paper/2308.08742/) (2023)
- [A Comprehensive Study of Knowledge Editing for Large Language Models](https://mlsanity.com/paper/2401.01286/) (2024)
- [Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue](https://mlsanity.com/paper/2401.04700/) (2024)
- [EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models](https://mlsanity.com/paper/2308.07269/) (2023)
- [Model Editing at Scale leads to Gradual and Catastrophic Forgetting](https://mlsanity.com/paper/2401.07453/) (2024)
- [Calibrating Factual Knowledge in Pretrained Language Models](https://mlsanity.com/paper/2210.03329/) (2022)
- [MELO: Enhancing Model Editing with Neuron-Indexed Dynamic LoRA](https://mlsanity.com/paper/2312.11795/) (2023)
- [Detecting Edit Failures In Large Language Models: An Improved Specificity Benchmark](https://mlsanity.com/paper/2305.17553/) (2023)
- [A Unified Framework for Model Editing](https://mlsanity.com/paper/2403.14236/) (2024)
- [The Butterfly Effect of Model Editing: Few Edits Can Trigger Large Language Models Collapse](https://mlsanity.com/paper/2402.09656/) (2024)
- [Rebuilding ROME : Resolving Model Collapse during Sequential Model Editing](https://mlsanity.com/paper/2403.07175/) (2024)
- [AKEW: Assessing Knowledge Editing in the Wild](https://mlsanity.com/paper/2402.18909/) (2024)

## Built on by (most cited first, 27 of 27)

- [CaKE: Circuit-aware Editing Enables Generalizable Knowledge Learners](https://mlsanity.com/paper/2503.16356/) (2025)
- [UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Language Models](https://mlsanity.com/paper/2505.14679/) (2025)
- [Fine-tuning Done Right in Model Editing](https://mlsanity.com/paper/2509.22072/) (2025)
- [Is Model Editing Built on Sand? Revealing Its Illusory Success and Fragile Foundation](https://mlsanity.com/paper/2510.00625/) (2025)
- [EvoEdit: Evolving Null-space Alignment for Robust and Efficient Knowledge Editing](https://mlsanity.com/paper/2510.13851/) (2025)
- [CrispEdit: Low-Curvature Projections for Scalable Non-Destructive LLM Editing](https://mlsanity.com/paper/2602.15823/) (2026)
- [Beyond Memorization: A Rigorous Evaluation Framework for Medical Knowledge Editing](https://mlsanity.com/paper/2506.03490/) (2025)
- [Are We Evaluating the Edit Locality of LLM Model Editing Properly?](https://mlsanity.com/paper/2601.17343/) (2026)
- [LyapLock: Bounded Knowledge Preservation in Sequential Large Language Model Editing](https://mlsanity.com/paper/2505.15702/) (2025)
- [InComeS: Integrating Compression and Selection Mechanisms into LLMs for Efficient Model Editing](https://mlsanity.com/paper/2505.22156/) (2025)
- [From Backward Spreading to Forward Replay: Revisiting Target Construction in LLM Parameter Editing](https://mlsanity.com/paper/2605.00358/) (2026)
- [EtCon: Edit-then-Consolidate for Reliable Knowledge Editing](https://mlsanity.com/paper/2512.04753/) (2025)
- [The Model Agreed, But Didn't Learn: Diagnosing Surface Compliance in Large Language Models](https://mlsanity.com/paper/2604.05995/) (2026)
- [BetaEdit: Null-Space Constrained Sequential Model Editing](https://mlsanity.com/paper/2605.09285/) (2026)
- [From Fact Overwriting to Knowledge Evolution: Causal Editing via On-Policy Self-Distillation](https://mlsanity.com/paper/2605.28303/) (2026)
- [Beyond Perplexity: A Behavioral Evaluation Framework for Deployment-Memory Claims in LLM Test-Time Training](https://mlsanity.com/paper/2607.00368/) (2026)
- [ADS-Edit: A Multimodal Knowledge Editing Dataset for Autonomous Driving Systems](https://mlsanity.com/paper/2503.20756/) (2025)
- [Benchmarking and Rethinking Knowledge Editing for Large Language Models](https://mlsanity.com/paper/2505.18690/) (2025)
- [Beyond Hidden-Layer Manipulation: Semantically-Aware Logit Interventions for Debiasing LLMs](https://mlsanity.com/paper/2510.23650/) (2025)
- [Detecting and Eliminating Neural Network Backdoors Through Active Paths with Application to Intrusion Detection](https://mlsanity.com/paper/2603.10641/) (2026)
- [Revisiting Parameter-Based Knowledge Editing in Large Language Models: Theoretical Limits and Empirical Evidence](https://mlsanity.com/paper/2606.00570/) (2026)
- [Evaluating and Understanding Model Editing for Medical Vision Language Models](https://mlsanity.com/paper/2607.05310/) (2026)
- [Hybrid-Policy Self-Editing for Composable Unstructured Knowledge Editing](https://mlsanity.com/paper/2608.11660/) (2026)
- [KLOD: Locality-Preserving Knowledge Editing via Non-Target Distribution Preservation](https://mlsanity.com/paper/2608.27839/) (2026)
- [Generalizable Lifelong Model Editing via Preference Optimization](https://mlsanity.com/paper/2609.36748/) (2026)
- [Improving Atomic-Fact Recall via Focused Views in Unstructured Knowledge Editing](https://mlsanity.com/paper/2610.02772/) (2026)
- [Mind the Gaps: From Failure Attribution to Closed-Form Repair of Code Language Models](https://mlsanity.com/paper/2610.05277/) (2026)

## Abstract (arXiv)

Despite near-perfect results reported in the literature, the effectiveness of model editing in real-world applications remains unclear. To bridge this gap, we introduce QAEdit, a new benchmark aligned with widely used question answering (QA) datasets, and WILD, a task-agnostic evaluation framework designed to better reflect real-world usage of model editing. Our single editing experiments show that current editing methods perform substantially worse than previously reported (38.5% vs. 96.8%). We demonstrate that it stems from issues in the synthetic evaluation practices of prior work. Among them, the most severe is the use of teacher forcing during testing, which leaks both content and length of the ground truth, leading to overestimated performance. Furthermore, we simulate practical deployment by sequential editing, revealing that current approaches fail drastically with only 1000 edits. This work calls for a shift in model editing research toward rigorous evaluation and the development of robust, scalable methods that can reliably update knowledge in LLMs for real-world use.