# MELO: Enhancing Model Editing with Neuron-Indexed Dynamic LoRA

arXiv: https://arxiv.org/abs/2312.11795 · HTML: https://mlsanity.com/paper/2312.11795/

## Key facts

- Cited by 80 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 19 Dec 2023.

## Builds on (most cited first, 19 of 19)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Memory-Based Model Editing at Scale](https://mlsanity.com/paper/2206.06520/) (2022)
- [Zero-Shot Relation Extraction via Reading Comprehension](https://mlsanity.com/paper/1706.04115/) (2017)
- [Editing Large Language Models: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2305.13172/) (2023)
- [Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors](https://mlsanity.com/paper/2211.11031/) (2022)
- [SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer](https://mlsanity.com/paper/2110.07904/) (2021)
- [DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation](https://mlsanity.com/paper/2210.07558/) (2022)
- [Transformer-Patcher: One Mistake worth One Neuron](https://mlsanity.com/paper/2301.09785/) (2023)
- [FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing](https://mlsanity.com/paper/2203.07228/) (2022)
- [On Continual Model Refinement in Out-of-Distribution Data Streams](https://mlsanity.com/paper/2205.02014/) (2022)

## Built on by (most cited first, 30 of 80)

- [AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models](https://mlsanity.com/paper/2410.02355/) (2024)
- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [Detoxifying Large Language Models via Knowledge Editing](https://mlsanity.com/paper/2403.14472/) (2024)
- [MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learning](https://mlsanity.com/paper/2402.11260/) (2024)
- [Neeko: Leveraging Dynamic LoRA for Efficient Multi-Character Role-Playing Agent](https://mlsanity.com/paper/2402.13717/) (2024)
- [Lifelong Knowledge Editing for LLMs with Retrieval-Augmented Continuous Prompt Learning](https://mlsanity.com/paper/2405.03279/) (2024)
- [Knowledge Boundary of Large Language Models: A Survey](https://mlsanity.com/paper/2412.12472/) (2024)
- [Position: Episodic Memory is the Missing Piece for Long-Term LLM Agents](https://mlsanity.com/paper/2502.06975/) (2025)
- [In-Context Editing: Learning Knowledge from Self-Induced Distributions](https://mlsanity.com/paper/2406.11194/) (2024)
- [Perturbation-Restrained Sequential Model Editing](https://mlsanity.com/paper/2405.16821/) (2024)
- [Can Knowledge Editing Really Correct Hallucinations?](https://mlsanity.com/paper/2410.16251/) (2024)
- [The Mirage of Model Editing: Revisiting Evaluation in the Wild](https://mlsanity.com/paper/2502.11177/) (2025)
- [Model Editing by Standard Fine-Tuning](https://mlsanity.com/paper/2402.11078/) (2024)
- [Editing Conceptual Knowledge for Large Language Models](https://mlsanity.com/paper/2403.06259/) (2024)
- [Neuron-Level Sequential Editing for Large Language Models](https://mlsanity.com/paper/2410.04045/) (2024)
- [Unlocking Efficient, Scalable, and Continual Knowledge Editing with Basis-Level Representation Fine-Tuning](https://mlsanity.com/paper/2503.00306/) (2025)
- [InstructEdit: Instruction-based Knowledge Editing for Large Language Models](https://mlsanity.com/paper/2402.16123/) (2024)
- [EVEDIT: Event-based Knowledge Editing with Deductive Editing Boundaries](https://mlsanity.com/paper/2402.11324/) (2024)
- [Has this Fact been Edited? Detecting Knowledge Edits in Language Models](https://mlsanity.com/paper/2405.02765/) (2024)
- [Attribution Analysis Meets Model Editing: Advancing Knowledge Correction in Vision Language Models with VisEdit](https://mlsanity.com/paper/2408.09916/) (2024)
- [Editing as Unlearning: Are Knowledge Editing Methods Strong Baselines for Large Language Model Unlearning?](https://mlsanity.com/paper/2505.19855/) (2025)
- [Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models](https://mlsanity.com/paper/2406.01436/) (2024)
- [Navigating the Dual Facets: A Comprehensive Evaluation of Sequential Memory Editing in Large Language Models](https://mlsanity.com/paper/2402.11122/) (2024)
- [Editing Factual Knowledge and Explanatory Ability of Medical Large Language Models](https://mlsanity.com/paper/2402.18099/) (2024)
- [MPN: Leveraging Multilingual Patch Neuron for Cross-lingual Model Editing](https://mlsanity.com/paper/2401.03190/) (2024)
- [UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Language Models](https://mlsanity.com/paper/2505.14679/) (2025)
- [Fine-tuning Done Right in Model Editing](https://mlsanity.com/paper/2509.22072/) (2025)
- [ChroKnowledge: Unveiling Chronological Knowledge of Language Models in Multiple Domains](https://mlsanity.com/paper/2410.09870/) (2024)
- [Mitigating Heterogeneous Token Overfitting in LLM Knowledge Editing](https://mlsanity.com/paper/2502.00602/) (2025)
- [Exploiting Edited Large Language Models as General Scientific Optimizers](https://mlsanity.com/paper/2503.09620/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have shown great success in various Natural Language Processing (NLP) tasks, whist they still need updates after deployment to fix errors or keep pace with the changing knowledge in the world. Researchers formulate such problem as Model Editing and have developed various editors focusing on different axes of editing properties. However, current editors can hardly support all properties and rely on heavy computational resources. In this paper, we propose a plug-in Model Editing method based on neuron-indexed dynamic LoRA (MELO), which alters the behavior of language models by dynamically activating certain LoRA blocks according to the index built in an inner vector database. Our method satisfies various editing properties with high efficiency and can be easily integrated into multiple LLM backbones. Experimental results show that our proposed MELO achieves state-of-the-art editing performance on three sequential editing tasks (document classification, question answering and hallucination correction), while requires the least trainable parameters and computational cost.