mlsanityData to 7 Oct 2026

Object Hallucination-Free Reinforcement Unlearning for Vision-Language Models

arXiv 2605.08031 ↗First posted 8 May 2026cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

also in 1 more
Abstract (arXiv)

Vision-language models (VLMs) raise growing concerns about privacy, copyright, and bias, motivating machine unlearning to remove sensitive knowledge. However, existing methods primarily fine-tune the language decoder, leading to superficial forgetting that fails to erase underlying visual representations and often introduces object hallucination. We propose HFRU, a reinforcement unlearning framework that operates on the vision encoder for deep semantic removal. Our two-stage approach combines alignment disruption with GRPO-based optimization using a composite reward, including an abstraction reward that encourages semantically valid substitutions and mitigates hallucinations. Experiments on object recognition and face identity tasks show that HFRU achieves over 98% forgetting and retention performance, while introducing negligible object hallucination, significantly outperforming prior methods.Our code and implementation details are available at https://github.com/XMUDeepLIT/HFRU.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 41 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

39 of 41 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projection18 Dec
  2. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb
  3. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  4. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  5. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 34

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  8. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  9. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  10. 2017Deeper, Broader and Artier Domain Generalizationcited by 983
  11. 2017VGGFace2: A dataset for recognising faces across pose and agecited by 911
  12. 2025OpenAI GPT-5 System Cardcited by 910
22 more
  1. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  2. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  3. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  4. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 484
  5. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  6. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  7. 2023Large Language Model Unlearningcited by 288
  8. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  9. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearningcited by 160
  10. 2023ProPILE: Probing Privacy Leakage in Large Language Modelscited by 127
  11. 2022Continual Learning and Private Unlearningcited by 124
  12. 2024Machine Unlearning of Pre-trained Large Language Modelscited by 118
  13. 2022Do Language Models Plagiarize?cited by 72
  14. 2025Reinforcement Learning from Human Feedbackcited by 53
  15. 2024UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Modelscited by 49
  16. 2024Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Modelscited by 44
  17. 2022How Large Language Models are Transforming Machine-Paraphrased Plagiarismcited by 38
  18. 2025Modality-Aware Neuron Pruning for Unlearning in Multimodal Large Language Modelscited by 25
  19. 2024Supervised Algorithmic Fairness in Distribution Shifts: A Surveycited by 23
  20. 2024Targeted Unlearning with Single Layer Unlearning Gradientcited by 16
  21. 2025SAUCE: Selective Concept Unlearning in Vision-Language Models with Sparse Autoencoderscited by 12
  22. 2026Bi-directional Bias Attribution: Debiasing Large Language Models without Modifying Promptscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026On the Robustness of Machine Unlearning for Vision-Language Modelscited by 1
  2. 2026ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Modelscited by 0
  3. 2026Open-MMUnlearning: Unifying Methods and Evaluation for MLLM Unlearningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Saying Less: Fine-Grained Alignment for Informative and Faithful Vision-Language Models
  2. 2025Analyzing and Mitigating Object Hallucination: A Training Bias Perspective
  3. 2026Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration
  4. 2026HIME: Mitigating Object Hallucinations in LVLMs via Hallucination Insensitivity Model Editing
  5. 2026NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors
  6. 2026Do Not Leave a Gap: Hallucination-Free Object Concealment in Vision-Language Models
  7. 2025Watch Closely: Mitigating Object Hallucinations in Large Vision-Language Models with Disentangled Decoding
  8. 2026HII-DPO: Eliminate Hallucination via Accurate Hallucination-Inducing Counterfactual Images