mlsanityData to 7 Oct 2026

Can We Edit Multimodal Large Language Models?

arXiv 2310.08475 ↗First posted 12 Oct 2023cs.CL cs.AI cs.CVcited by 44

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

In this paper, we focus on editing Multimodal Large Language Models (MLLMs). Compared to editing single-modal LLMs, multimodal model editing is more challenging, which demands a higher level of scrutiny and careful consideration in the editing process. To facilitate research in this area, we construct a new benchmark, dubbed MMEdit, for editing multimodal LLMs and establishing a suite of innovative metrics for evaluation. We conduct comprehensive experiments involving various model editing baselines and analyze the impact of editing different components for multimodal LLMs. Empirically, we notice that previous baselines can implement editing multimodal LLMs to some extent, but the effect is still barely satisfactory, indicating the potential difficulty of this task. We hope that our work can provide the NLP community with insights. Code and dataset are available in https://github.com/zjunlp/EasyEdit.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 69 references that are in the map. First posted on arXiv 12 Oct 2023.

Builds on

49 of 69 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Models19 Sep
  2. 2023A Survey on Multimodal Large Language Models23 Jun
  3. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May
  4. 2023VPGTrans: Transfer Visual Prompt Generator across LLMs2 May

Other references · 45

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022Generative Adversarial Networkscited by 5,603
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  11. 2015VQA: Visual Question Answeringcited by 2,489
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
33 more
  1. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  2. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  3. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  4. 2022Editing Models with Task Arithmeticcited by 1,076
  5. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  6. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  7. 2022Mass-Editing Memory in a Transformercited by 976
  8. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  9. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  10. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  11. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  12. 2021Editing Factual Knowledge in Language Modelscited by 548
  13. 2023Erasing Concepts from Diffusion Modelscited by 530
  14. 2021Fast Model Editing at Scalecited by 491
  15. 2022Memory-Based Model Editing at Scalecited by 443
  16. 2022Multimodal Learning with Transformers: A Surveycited by 399
  17. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  18. 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 292
  19. 2023Can We Edit Factual Knowledge by In-Context Learning?cited by 284
  20. 2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 267
  21. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 238
  22. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
  23. 2023Transformer-Patcher: One Mistake worth One Neuroncited by 196
  24. 2019Image Captioning: Transforming Objects into Wordscited by 171
  25. 2023PMET: Precise Model Editing in a Transformercited by 170
  26. 2020Editable Neural Networkscited by 154
  27. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 140
  28. 2023EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Modelscited by 97
  29. 2022Calibrating Factual Knowledge in Pretrained Language Modelscited by 92
  30. 2023Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledgecited by 66
  31. 2023Unveiling the Pitfalls of Knowledge Editing for Large Language Modelscited by 43
  32. 2022Language Anisotropic Cross-Lingual Model Editingcited by 26
  33. 2023Cross-Lingual Knowledge Editing in Large Language Modelscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 17; 2025: 16; 2026: 9.

44 papers.

Sort:

By area:

  1. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  2. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  3. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
  4. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  5. 2024Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescuecited by 101
  6. 2024MyVLM: Personalizing VLMs for User-Specific Queriescited by 58
  7. 2024Learning to Edit: Aligning LLMs with Knowledge Editingcited by 43
  8. 2024VLKEB: A Large Vision-Language Model Knowledge Editing Benchmarkcited by 30
  9. 2023Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applicationscited by 28
  10. 2025DUNE: Distilling a Universal Encoder from Heterogeneous 2D and 3D Teacherscited by 28
  11. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  12. 2025MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual Knowledgecited by 22
32 more
  1. 2024Everything is Editable: Extend Knowledge Editing to Unstructured Data in Large Language Modelscited by 18
  2. 2024InstructEdit: Instruction-based Knowledge Editing for Large Language Modelscited by 17
  3. 2024Lifelong Knowledge Editing for Vision Language Models with Low-Rank Mixture-of-Expertscited by 14
  4. 2025What Is That Talk About? A Video-to-Text Summarization Dataset for Scientific Presentationscited by 11
  5. 2025Unlearning Sensitive Information in Multimodal LLMs: Benchmark and Attack-Defense Evaluationcited by 11
  6. 2024Neighboring Perturbations of Knowledge Editing on Large Language Modelscited by 10
  7. 2024Locating and Mitigating Gender Bias in Large Language Modelscited by 10
  8. 2024Visual-Oriented Fine-Grained Knowledge Editing for MultiModal Large Language Modelscited by 9
  9. 2025BalancEdit: Dynamically Balancing the Generality-Locality Trade-off in Multi-modal Model Editingcited by 7
  10. 2025MINED: Probing and Updating with Multimodal Time-Sensitive Knowledge for Large Multimodal Modelscited by 7
  11. 2025OFFSIDE: Benchmarking Unlearning Misinformation in Multimodal Large Language Modelscited by 7
  12. 2025CLIP-UP: CLIP-Based Unanswerable Problem Detection for Visual Question Answeringcited by 6
  13. 2025When Large Multimodal Models Confront Evolving Knowledge: Challenges and Explorationscited by 6
  14. 2025DualEdit: Dual Editing for Knowledge Updating in Vision-Language Modelscited by 5
  15. 2025SAKE: Towards Editing Auditory Attribute Knowledge of Large Audio-Language Modelscited by 5
  16. 2024Editing Knowledge Representation of Language Model via Rephrased Prefix Promptscited by 4
  17. 2024Who Brings the Frisbee: Probing Hidden Hallucination Factors in Large Vision-Language Model via Causality Analysiscited by 4
  18. 2024ComprehendEdit: A Comprehensive Dataset and Evaluation Framework for Multimodal Knowledge Editingcited by 4
  19. 2025MolEdit: Knowledge Editing for Multimodal Molecule Language Modelscited by 3
  20. 2025Efficient Few-Shot Continual Learning in Vision-Language Modelscited by 2
  21. 2025MemEIC: A Step Toward Continual and Compositional Knowledge Editingcited by 2
  22. 2025EvoEdit: Lifelong Free-Text Knowledge Editing through Latent Perturbation Augmentation and Knowledge-driven Parameter Fusioncited by 2
  23. 2026Echoes within the Reasoning: Stealthy and Effective Watermarking via Chain of Thoughtcited by 2
  24. 2025Towards Meta-Cognitive Knowledge Editing for Multimodal LLMscited by 1
  25. 2026Generalizable Multimodal Large Language Model Editing via Invariant Trajectory Learningcited by 1
  26. 2026DSCA: Dynamic Subspace Concept Alignment for Lifelong VLM Editingcited by 1
  27. 2026Modality-Decoupled Online Recursive Editingcited by 1
  28. 2026ReasonEdit: Editing Vision-Language Models using Human Reasoningcited by 0
  29. 2026Uncovering Entity Identity Confusion in Multimodal Knowledge Editingcited by 0
  30. 2026CrossCult-KIBench: A Benchmark for Cross-Cultural Knowledge Insertion in MLLMscited by 0
  31. 2026Beyond Binary Edits Robust Multimodal Knowledge Editing with Adversarial Subspace Alignmentcited by 0
  32. 2026Multimodal Knowledge Edit-Scoped Generalization for Online Recursive MLLM Editingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Self-Improvement in Multimodal Large Language Models: A Survey
  2. 2024MC-MKE: A Fine-Grained Multimodal Knowledge Editing Benchmark Emphasizing Modality Consistency
  3. 2024MM-LLMs: Recent Advances in MultiModal Large Language Models
  4. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspective
  5. 2024LLMs Meet Multimodal Generation and Editing: A Survey
  6. 2024Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges
  7. 2024The Butterfly Effect of Model Editing: Few Edits Can Trigger Large Language Models Collapse
  8. 2024Personalized Multimodal Large Language Models: A Survey