mlsanityData to 14 Oct 2026

Before Forgetting, Learn to Remember: Revisiting Foundational Learning Failures in LVLM Unlearning Benchmarks

arXiv 2605.03759 ↗First posted 5 May 2026cs.CV cs.AIcited by 3

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

While Large Vision-Language Models (LVLMs) offer powerful capabilities, they pose privacy risks by unintentionally memorizing sensitive personal information. Current unlearning benchmarks attempt to mitigate this using fictitious identities but overlook a critical stage 1 failure: models fail to effectively memorize target information initially, rendering subsequent unlearning evaluations unreliable. Diagnosing under-memorization and the multi-hop curse as root causes, we introduce ReMem, a Reliable Multi-hop and Multi-image Memorization Benchmark. ReMem ensures robust foundational learning through principled data scaling, reasoning-aware QA pairs, and diverse visual contexts. Additionally, we propose a novel Exposure metric to quantify the depth of information erasure from the model's internal probability distribution. Extensive experiments demonstrate that ReMem provides a rigorous and trustworthy framework for diagnosing both learning and unlearning behaviors in LVLMs.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 43 references that are in the map. First posted on arXiv 5 May 2026.

Builds on

27 of 43 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench29 Oct

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  6. 2020Extracting Training Data from Large Language Modelscited by 2,189
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  8. 2019Machine Unlearningcited by 1,061
  9. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  10. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  11. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 533
  12. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 504
14 more
  1. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 437
  2. 2021Unrolling SGD: Understanding Factors Influencing Machine Unlearningcited by 279
  3. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearningcited by 172
  4. 2025How much do language models memorize?cited by 61
  5. 2024Understanding Information Storage and Transfer in Multi-modal Large Language Modelscited by 49
  6. 2024CLEAR: Character Unlearning in Textual and Visual Modalitiescited by 39
  7. 2021Decomposing Complex Questions Makes Multi-Hop QA Easier and More Interpretablecited by 35
  8. 2025The European Union general data protection regulation: what it is and what it meanscited by 31
  9. 2024VLKEB: A Large Vision-Language Model Knowledge Editing Benchmarkcited by 30
  10. 2025Decoupled Distillation to Erase: A General Unlearning Method for Any Class-centric Taskscited by 28
  11. 2022Modeling Multi-hop Question Answering as Single Sequence Predictioncited by 21
  12. 2025Unlearning Isn't Invisible: Detecting Unlearning Traces in LLMs from Model Outputscited by 11
  13. 2025Zero-Shot Complex Question-Answering on Long Scientific Documentscited by 1
  14. 2025Knowledge Editing for Multi-Hop Question Answering Using Semantic Analysiscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Does Forgetting Transfer Across Modalities? A Real-World Benchmark for Cross-Modal Knowledge Unlearning Evaluationcited by 1
  2. 2026AIM: Anchor Identity Features, Then Match for Multimodal Large Language Model Unlearningcited by 1
  3. 2026Where Identity Lives: Localized, Retain-Free Identity Unlearning in Multimodal Large Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026On the Robustness of Machine Unlearning for Vision-Language Models
  2. 2025LLM Unlearning on Noisy Forget Sets: A Study of Incomplete, Rewritten, and Watermarked Data
  3. 2025Towards Benign Memory Forgetting for Selective Multimodal Large Language Model Unlearning
  4. 2026Visual-Noise Guided In-Context Distillation for Multimodal Large Language Model Unlearning
  5. 2026Toward Fine-Grained Forgetting:Attribute Unlearning for Multimodal Large Language Models
  6. 2025BLUR: A Benchmark for LLM Unlearning Robust to Forget-Retain Overlap
  7. 2026Suppression Is Not Forgetting: Residual Recoverability in Visual Concept Unlearning for VLMs
  8. 2026PPU-Bench:Real World Benchmark for Personalized Partial Unlearning in Vision Language Models