mlsanityData to 14 Oct 2026

Partition-Aware Unlearning for Removing Spurious Correlations in Large Vision-Language Models

arXiv 2608.29996 ↗First posted 30 Aug 2026cs.CV cs.AI cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

also in 1 more
Abstract (arXiv)

Large Vision-Language Models (LVLMs) achieve strong performance across many multimodal tasks; however, they often exploit spurious object-background correlations, resulting in predictions driven by contextual shortcuts rather than object-relevant visual evidence. Despite growing interest in hallucination and robustness evaluation, existing benchmarks provide limited control over whether model predictions are grounded in the target object or induced by correlated background cues. In this work, we introduce PURGE (underlinePartition-aware underlineUnlearning for underlineRemoving spurious-correlation underlineGenerated underlineErrors), a framework for constructing, benchmarking, and mitigating spurious-correlation-induced failures in LVLMs. The framework consists of: -- (1) Structured dataset construction wherein we develop three complementary structured data construction strategies that partition examples by object-relevant evidence and spurious background cues, enabling controlled diagnosis of shortcut reliance; and -- (2) Partition-aware unlearning, which uses these partitions to selectively remove spurious object-background associations while preserving object-based reasoning. We evaluate the algo~framework across multiple LVLMs, including LLaVA-1.6-7B, Qwen3-VL-8B-Instruct, and Qwen3.5-9B, together with CLIP as a vision-language encoder, on a diverse suite of benchmarks, including CHAIR, POPE, Causal-HalBench, MM-SpuBench, AMBER, MMHal, and Waterbirds. Our results show that PURGE consistently reduces hallucinations and spurious-correlation-driven errors while maintaining or improving overall performance in most evaluated settings, providing both a reusable evaluation protocol and an effective mitigation framework for more reliable LVLMs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 53 references that are in the map. First posted on arXiv 30 Aug 2026.

Builds on

45 of 53 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Causal-HalBench: Uncovering LVLMs Object Hallucinations Through Causal Intervention13 Nov
  2. 2025Causal-LLaVA: Causal Disentanglement for Mitigating Hallucination in Multimodal Large Language Models26 May
  3. 2025SpurLens: Automatic Detection of Spurious Cues in Multimodal LLMs11 Mar
  4. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  5. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  6. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  7. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  8. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  8. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  9. 2019Invariant Risk Minimizationcited by 1,854
  10. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  11. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 738
  12. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
25 more
  1. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  2. 2018Object Hallucination in Image Captioningcited by 669
  3. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 504
  4. 2023Scaling Open-Vocabulary Object Detectioncited by 343
  5. 2023Debiasing Vision-Language Models via Biased Promptscited by 149
  6. 2018Image Inpainting via Generative Multi-column Convolutional Neural Networkscited by 81
  7. 2021Let there be a clock on the beach: Reducing Object Hallucination in Image Captioningcited by 76
  8. 2023DeAR: Debiasing Vision-Language Models with Additive Residualscited by 70
  9. 2025Vision Language Models are Biasedcited by 66
  10. 2023Mitigating Spurious Correlations in Multi-modal Models during Fine-tuningcited by 47
  11. 2024Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Modelscited by 46
  12. 2024MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMscited by 33
  13. 2025SafeEraser: Enhancing Safety in Multimodal Large Language Models through Multimodal Machine Unlearningcited by 32
  14. 2025MMUnlearner: Reformulating Multimodal Machine Unlearning in the Era of Multimodal Large Language Modelscited by 30
  15. 2024RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned Vision-Language Modelscited by 20
  16. 2024A Unified Debiasing Approach for Vision-Language Models across Modalities and Taskscited by 18
  17. 2024BendVLM: Test-Time Debiasing of Vision-Language Embeddingscited by 17
  18. 2025Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-Tuning and Can Be Mitigated by Machine Unlearningcited by 16
  19. 2025Escaping the SpuriVerse: Can Large Vision-Language Models Generalize Beyond Seen Spurious Correlations?cited by 14
  20. 2025SAUCE: Selective Concept Unlearning in Vision-Language Models with Sparse Autoencoderscited by 13
  21. 2025Approximate Domain Unlearning for Vision-Language Modelscited by 8
  22. 2026Interpretable Debiasing of Vision-Language Models for Social Fairnesscited by 6
  23. 2024Learning Invariant Causal Mechanism from Vision-Language Modelscited by 3
  24. 2025Cross-Modal Attention Guided Unlearning in Vision-Language Modelscited by 1
  25. 2026A Closed-Form Solution for Debiasing Vision-Language Models with Utility Guarantees Across Modalities and Taskscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Analyzing and Mitigating Object Hallucination: A Training Bias Perspective
  2. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models
  3. 2026TruthLens: Object Hallucination Detection via Self-Evaluating Truthfulness Scores in LVLMs
  4. 2026UniProbe: A Learnable Token-Level Hallucination Detector for Large VLMs using Multi-Structural Internal Representations
  5. 2023Mitigating Hallucination in Visual Language Models with Visual Supervision
  6. 2026Object Hallucination-Free Reinforcement Unlearning for Vision-Language Models
  7. 2025Evaluating Hallucination in Large Vision-Language Models based on Context-Aware Object Similarities
  8. 2026Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration