# VLKEB: A Large Vision-Language Model Knowledge Editing Benchmark

arXiv: https://arxiv.org/abs/2403.07350 · HTML: https://mlsanity.com/paper/2403.07350/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Image editing and quality assessment, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 28 of its 37 references that are in the map.
- First posted on arXiv 12 Mar 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Image editing and quality assessment](https://mlsanity.com/mllm/image-editing-and-quality-assessment/)

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration](https://mlsanity.com/paper/2311.04257/) (2023)
- [Fast Model Editing at Scale](https://mlsanity.com/paper/2110.11309/) (2021)
- [Memory-Based Model Editing at Scale](https://mlsanity.com/paper/2206.06520/) (2022)
- [Zero-Shot Relation Extraction via Reading Comprehension](https://mlsanity.com/paper/1706.04115/) (2017)
- [Editing Large Language Models: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2305.13172/) (2023)
- [MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions](https://mlsanity.com/paper/2305.14795/) (2023)
- [Can We Edit Factual Knowledge by In-Context Learning?](https://mlsanity.com/paper/2305.12740/) (2023)
- [Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors](https://mlsanity.com/paper/2211.11031/) (2022)
- [Evaluating the Ripple Effects of Knowledge Editing in Language Models](https://mlsanity.com/paper/2307.12976/) (2023)
- [Transformer-Patcher: One Mistake worth One Neuron](https://mlsanity.com/paper/2301.09785/) (2023)
- [Knowledge Editing for Large Language Models: A Survey](https://mlsanity.com/paper/2310.16218/) (2023)
- [A Comprehensive Study of Knowledge Editing for Large Language Models](https://mlsanity.com/paper/2401.01286/) (2024)
- [MMKG: Multi-Modal Knowledge Graphs](https://mlsanity.com/paper/1903.05485/) (2019)
- [EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models](https://mlsanity.com/paper/2308.07269/) (2023)
- [Calibrating Factual Knowledge in Pretrained Language Models](https://mlsanity.com/paper/2210.03329/) (2022)
- [Massive Editing for Large Language Models via Meta Learning](https://mlsanity.com/paper/2311.04661/) (2023)
- [Can We Edit Multimodal Large Language Models?](https://mlsanity.com/paper/2310.08475/) (2023)
- [WikiReading: A Novel Large-scale Language Understanding Task over Wikipedia](https://mlsanity.com/paper/1608.03542/) (2016)

## Built on by (most cited first, 30 of 30)

- [MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs](https://mlsanity.com/paper/2411.15296/) (2024)
- [Knowledge Mechanisms in Large Language Models: A Survey and Perspective](https://mlsanity.com/paper/2407.15017/) (2024)
- [Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts](https://mlsanity.com/paper/2505.17127/) (2025)
- [Can Knowledge Editing Really Correct Hallucinations?](https://mlsanity.com/paper/2410.16251/) (2024)
- [A Survey on Multimodal Benchmarks: In the Era of Large AI Models](https://mlsanity.com/paper/2409.18142/) (2024)
- [MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual Knowledge](https://mlsanity.com/paper/2502.19870/) (2025)
- [Can Editing LLMs Inject Harm?](https://mlsanity.com/paper/2407.20224/) (2024)
- [Lifelong Knowledge Editing for Vision Language Models with Low-Rank Mixture-of-Experts](https://mlsanity.com/paper/2411.15432/) (2024)
- [CLIP-UP: CLIP-Based Unanswerable Problem Detection for Visual Question Answering](https://mlsanity.com/paper/2501.01371/) (2025)
- [When Large Multimodal Models Confront Evolving Knowledge: Challenges and Explorations](https://mlsanity.com/paper/2505.24449/) (2025)
- [MINED: Probing and Updating with Multimodal Time-Sensitive Knowledge for Large Multimodal Models](https://mlsanity.com/paper/2510.19457/) (2025)
- [SAKE: Towards Editing Auditory Attribute Knowledge of Large Audio-Language Models](https://mlsanity.com/paper/2510.16917/) (2025)
- [MMKU-Bench: A Multimodal Update Benchmark for Diverse Visual Knowledge](https://mlsanity.com/paper/2603.15117/) (2026)
- [ComprehendEdit: A Comprehensive Dataset and Evaluation Framework for Multimodal Knowledge Editing](https://mlsanity.com/paper/2412.12821/) (2024)
- [MolEdit: Knowledge Editing for Multimodal Molecule Language Models](https://mlsanity.com/paper/2511.12770/) (2025)
- [Before Forgetting, Learn to Remember: Revisiting Foundational Learning Failures in LVLM Unlearning Benchmarks](https://mlsanity.com/paper/2605.03759/) (2026)
- [SNS-Bench-VL: Benchmarking Multimodal Large Language Models in Social Networking Services](https://mlsanity.com/paper/2505.23065/) (2025)
- [MedMKEB: A Comprehensive Knowledge Editing Benchmark for Medical Multimodal Large Language Models](https://mlsanity.com/paper/2508.05083/) (2025)
- [MemEIC: A Step Toward Continual and Compositional Knowledge Editing](https://mlsanity.com/paper/2510.25798/) (2025)
- [Towards Meta-Cognitive Knowledge Editing for Multimodal LLMs](https://mlsanity.com/paper/2509.05714/) (2025)
- [Generalizable Multimodal Large Language Model Editing via Invariant Trajectory Learning](https://mlsanity.com/paper/2601.19700/) (2026)
- [DSCA: Dynamic Subspace Concept Alignment for Lifelong VLM Editing](https://mlsanity.com/paper/2604.07965/) (2026)
- [Uncovering Entity Identity Confusion in Multimodal Knowledge Editing](https://mlsanity.com/paper/2605.06096/) (2026)
- [ADS-Edit: A Multimodal Knowledge Editing Dataset for Autonomous Driving Systems](https://mlsanity.com/paper/2503.20756/) (2025)
- [Lifting Data-Tracing Machine Unlearning to Knowledge-Tracing for Foundation Models](https://mlsanity.com/paper/2506.11253/) (2025)
- [Uncovering and Mitigating Transient Blindness in Multimodal Model Editing](https://mlsanity.com/paper/2511.13243/) (2025)
- [ReasonEdit: Editing Vision-Language Models using Human Reasoning](https://mlsanity.com/paper/2602.02408/) (2026)
- [CrossCult-KIBench: A Benchmark for Cross-Cultural Knowledge Insertion in MLLMs](https://mlsanity.com/paper/2605.06115/) (2026)
- [Beyond Binary Edits Robust Multimodal Knowledge Editing with Adversarial Subspace Alignment](https://mlsanity.com/paper/2605.23780/) (2026)
- [Multimodal Knowledge Edit-Scoped Generalization for Online Recursive MLLM Editing](https://mlsanity.com/paper/2607.01978/) (2026)

## Abstract (arXiv)

Recently, knowledge editing on large language models (LLMs) has received considerable attention. Compared to this, editing Large Vision-Language Models (LVLMs) faces extra challenges from diverse data modalities and complicated model components, and data for LVLMs editing are limited. The existing LVLM editing benchmark, which comprises three metrics (Reliability, Locality, and Generality), falls short in the quality of synthesized evaluation images and cannot assess whether models apply edited knowledge in relevant content. Therefore, we employ more reliable data collection methods to construct a new Large Vision-Language Model Knowledge Editing Benchmark, VLKEB, and extend the Portability metric for more comprehensive evaluation. Leveraging a multi-modal knowledge graph, our image data are bound with knowledge entities. This can be further used to extract entity-related knowledge, which constitutes the base of editing data. We conduct experiments of different editing methods on five LVLMs, and thoroughly analyze how do they impact the models. The results reveal strengths and deficiencies of these methods and hopefully provide insights for future research. The codes and dataset are available at: https://github.com/VLKEB/VLKEB.