mlsanityData to 7 Oct 2026

Towards Unified Multimodal Editing with Enhanced Knowledge Collaboration

arXiv 2409.19872 ↗First posted 30 Sep 2024cs.CVcited by 18

On the map: Multimodal LLMs / vision-language models → Image editing and quality assessment

Abstract (arXiv)

The swift advancement in Multimodal LLMs (MLLMs) also presents significant challenges for effective knowledge editing. Current methods, including intrinsic knowledge editing and external knowledge resorting, each possess strengths and weaknesses, struggling to balance the desired properties of reliability, generality, and locality when applied to MLLMs. In this paper, we propose UniKE, a novel multimodal editing method that establishes a unified perspective and paradigm for intrinsic knowledge editing and external knowledge resorting. Both types of knowledge are conceptualized as vectorized key-value memories, with the corresponding editing processes resembling the assimilation and accommodation phases of human cognition, conducted at the same semantic levels. Within such a unified framework, we further promote knowledge collaboration by disentangling the knowledge representations into the semantic and truthfulness spaces. Extensive experiments validate the effectiveness of our method, which ensures that the post-edit MLLM simultaneously maintains excellent reliability, generality, and locality. The code for UniKE is available at urlhttps://github.com/beepkh/UniKE.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Image editing and quality assessment, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 51 references that are in the map. First posted on arXiv 30 Sep 2024.

Builds on

39 of 51 references.

References · 39

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  8. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  9. 2022Mass-Editing Memory in a Transformercited by 976
  10. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  11. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  12. 2021Editing Factual Knowledge in Language Modelscited by 548
27 more
  1. 2021Fast Model Editing at Scalecited by 491
  2. 2022Memory-Based Model Editing at Scalecited by 443
  3. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  4. 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 292
  5. 2023Can We Edit Factual Knowledge by In-Context Learning?cited by 284
  6. 2023Transformer-Patcher: One Mistake worth One Neuroncited by 196
  7. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  8. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognitioncited by 151
  9. 2017Generalization in Deep Learningcited by 146
  10. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Datacited by 124
  11. 2023Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructionscited by 76
  12. 2022Fine-Grained Semantically Aligned Vision-Language Pre-Trainingcited by 68
  13. 2022Compositional Temporal Grounding with Structured Variational Cross-Graph Correspondence Learningcited by 59
  14. 2024Towards Semantic Equivalence of Tokenization in Multimodal LLMcited by 45
  15. 2022Decoupling Knowledge from Memorization: Retrieval-augmented Prompt Learningcited by 36
  16. 2023Finding and Editing Multi-Modal Neurons in Pre-Trained Transformerscited by 36
  17. 2023SGL-PT: A Strong Graph Learner with Graph Prompt Tuningcited by 35
  18. 2023Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open Worldcited by 35
  19. 2023GraphControl: Adding Conditional Control to Universal Graph Pre-trained Models for Graph Domain Transfer Learningcited by 32
  20. 2024Enhancing Video-Language Representations with Structural Spatio-Temporal Alignmentcited by 32
  21. 2024Auto-Encoding Morph-Tokens for Multimodal LLMcited by 30
  22. 2023Variational Cross-Graph Reasoning and Adaptive Structured Semantics Learning for Compositional Temporal Groundingcited by 19
  23. 2023Self-supervised Meta-Prompt Learning with Meta-Gradient Regularization for Few-shot Generalizationcited by 15
  24. 2024Non-confusing Generation of Customized Concepts in Diffusion Modelscited by 15
  25. 2022RoSA: A Robust Self-Aligned Framework for Node-Node Graph Contrastive Learningcited by 12
  26. 2023Interactive Data Synthesis for Systematic Vision Adaptation via LLMs-AIGCs Collaborationcited by 12
  27. 2023Improving Vision Anomaly Detection with the Guidance of Language Modalitycited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 11; 2026: 4.

18 papers.

Sort:

By area:

  1. 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 235
  2. 2025Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokenscited by 27
  3. 2024Unified Generative and Discriminative Training for Multi-modal Large Language Modelscited by 15
  4. 2025Position: Editing Large Language Models Poses Serious Safety Riskscited by 15
  5. 2025FocusDiff: Advancing Fine-Grained Text-Image Alignment for Autoregressive Visual Generation through RLcited by 13
  6. 2024Mastering Collaborative Multi-modal Data Selection: A Focus on Informativeness, Uniqueness, and Representativenesscited by 10
  7. 2025WiseEdit: Benchmarking Cognition- and Creativity-Informed Image Editingcited by 10
  8. 2025Boosting Virtual Agent Learning and Reasoning: A Step-Wise, Multi-Dimensional, and Generalist Reward Model with Benchmarkcited by 9
  9. 2025Janus-Pro-R1: Advancing Collaborative Visual Comprehension and Generation via Reinforcement Learningcited by 8
  10. 2025What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilitiescited by 8
  11. 2025Focusing by Contrastive Attention: Enhancing VLMs' Visual Reasoningcited by 8
  12. 2025Towards Meta-Cognitive Knowledge Editing for Multimodal LLMscited by 1
6 more
  1. 2025OmniMoGen: Unifying Human Motion Generation via Learning from Interleaved Text-Motion Instructionscited by 1
  2. 2026Generalizable Multimodal Large Language Model Editing via Invariant Trajectory Learningcited by 1
  3. 2026Correct When Paired, Wrong When Split: Decoupling and Editing Modality-Specific Neurons in MLLMscited by 1
  4. 2025ADS-Edit: A Multimodal Knowledge Editing Dataset for Autonomous Driving Systemscited by 0
  5. 2026Uncovering Entity Identity Confusion in Multimodal Knowledge Editingcited by 0
  6. 2026Beyond Binary Edits Robust Multimodal Knowledge Editing with Adversarial Subspace Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024ComprehendEdit: A Comprehensive Dataset and Evaluation Framework for Multimodal Knowledge Editing
  2. 2024MC-MKE: A Fine-Grained Multimodal Knowledge Editing Benchmark Emphasizing Modality Consistency
  3. 2025MemEIC: A Step Toward Continual and Compositional Knowledge Editing
  4. 2026Multimodal Knowledge Edit-Scoped Generalization for Online Recursive MLLM Editing
  5. 2025MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual Knowledge
  6. 2026Towards Localized and Disentangled Knowledge Editing for Multimodal Large Language Models
  7. 2024Visual-Oriented Fine-Grained Knowledge Editing for MultiModal Large Language Models
  8. 2025KORE: Enhancing Knowledge Injection for Large Multimodal Models via Knowledge-Oriented Controls