mlsanityData to 14 Oct 2026

Fast Model Editing at Scale

arXiv 2110.11309 ↗First posted 21 Oct 2021cs.LG cs.AI cs.CLcited by 498

Not on the map. Why?

Cited by 46 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

While large pre-trained models have enabled impressive results on a variety of downstream tasks, the largest existing models still make errors, and even accurate predictions may become outdated over time. Because detecting all such failures at training time is impossible, enabling both developers and end users of such models to correct inaccurate outputs while leaving the model otherwise intact is desirable. However, the distributed, black-box nature of the representations learned by large neural networks makes producing such targeted edits difficult. If presented with only a single problematic input and new desired output, fine-tuning approaches tend to overfit; other editing algorithms are either computationally infeasible or simply ineffective when applied to very large models. To enable easy post-hoc editing at scale, we propose Model Editor Networks using Gradient Decomposition (MEND), a collection of small auxiliary editing networks that use a single desired input-output pair to make fast, local edits to a pre-trained model's behavior. MEND learns to transform the gradient obtained by standard fine-tuning, using a low-rank decomposition of the gradient to make the parameterization of this transformation tractable. MEND can be trained on a single GPU in less than a day even for 10 billion+ parameter models; once trained MEND enables rapid application of new edits to the pre-trained model. Our experiments with T5, GPT, BERT, and BART models show that MEND is the only approach to model editing that effectively edits the behavior of models with more than 10 billion parameters. Code and data available at https://sites.google.com/view/mend-editing.

Key facts. Cited by 498 papers in the map (data through 14 Oct 2026). Builds on 35 of its 57 references that are in the map. First posted on arXiv 21 Oct 2021.

Builds on

35 of 57 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2016Densely Connected Convolutional Networkscited by 6,720
  9. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  10. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  11. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  12. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
23 more
  1. 2019Language Models as Knowledge Bases?cited by 1,936
  2. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  3. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  4. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  5. 2016HyperNetworkscited by 1,027
  6. 2019How Can We Know What Language Models Know?cited by 959
  7. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  8. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  9. 2021Editing Factual Knowledge in Language Modelscited by 553
  10. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
  11. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 439
  12. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 426
  13. 2017Meta-SGD: Learning to Learn Quickly for Few-Shot Learningcited by 379
  14. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 352
  15. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 319
  16. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 230
  17. 2020Modifying Memories in Transformer Modelscited by 224
  18. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 166
  19. 2020Editable Neural Networkscited by 155
  20. 2019Meta-Learning with Warped Gradient Descentcited by 98
  21. 2019Generalized Inner Loop Meta-Learningcited by 93
  22. 2019Meta-Curvaturecited by 70
  23. 2021Provable Repair of Deep Neural Networkscited by 42

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 43; 2023: 84; 2024: 135; 2025: 117; 2026: 117.

498 papers.

Sort:

By area:

  1. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  2. 2022Editing Models with Task Arithmeticcited by 1,120
  3. 2022Mass-Editing Memory in a Transformercited by 1,001
  4. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 668
  5. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
  6. 2023Erasing Concepts from Diffusion Modelscited by 539
  7. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  8. 2022Memory-Based Model Editing at Scalecited by 447
  9. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 407
  10. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 372
  11. 2024Improving Alignment and Robustness with Circuit Breakerscited by 322
  12. 2023Ablating Concepts in Text-to-Image Diffusion Modelscited by 317
38 more
  1. 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 300
  2. 2022Post-hoc Concept Bottleneck Modelscited by 293
  3. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  4. 2023Can We Edit Factual Knowledge by In-Context Learning?cited by 285
  5. 2024Knowledge Conflicts for LLMs: A Surveycited by 283
  6. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 281
  7. 2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 271
  8. 2022Prompting GPT-3 To Be Reliablecited by 260
  9. 2022Reasoning with Language Model Prompting: A Surveycited by 252
  10. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 243
  11. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 242
  12. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 237
  13. 2024AlphaEdit: Null-Space Constrained Knowledge Editing for Language Modelscited by 236
  14. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  15. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMscited by 215
  16. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 206
  17. 2023Continual Pre-training of Language Modelscited by 198
  18. 2023Transformer-Patcher: One Mistake worth One Neuroncited by 197
  19. 2022Patching open-vocabulary models by interpolating weightscited by 190
  20. 2022Can Foundation Models Wrangle Your Data?cited by 175
  21. 2023PMET: Precise Model Editing in a Transformercited by 173
  22. 2022Large Language Models with Controllable Working Memorycited by 169
  23. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 163
  24. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 158
  25. 2022A Review on Language Models as Knowledge Basescited by 157
  26. 2023Taken out of context: On measuring situational awareness in LLMscited by 143
  27. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 141
  28. 2025MemOS: A Memory OS for AI Systemcited by 138
  29. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
  30. 2023Factuality Challenges in the Era of Large Language Modelscited by 126
  31. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 125
  32. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  33. 2023RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputscited by 105
  34. 2024Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescuecited by 103
  35. 2022Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completioncited by 102
  36. 2024BadEdit: Backdooring large language models by model editingcited by 96
  37. 2024Model Editing at Scale leads to Gradual and Catastrophic Forgettingcited by 93
  38. 2024MEMORYLLM: Towards Self-Updatable Large Language Modelscited by 93

50 most cited of 498 All 498 (JSON)