mlsanityData to 14 Oct 2026

PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Training

arXiv 2503.06486 ↗First posted 9 Mar 2025cs.CV cs.AIcited by 38

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

This paper aims to address the challenge of hallucinations in Multimodal Large Language Models (MLLMs) particularly for dense image captioning tasks. To tackle the challenge, we identify the current lack of a metric that finely measures the caption quality in concept level. We hereby introduce HalFscore, a novel metric built upon the language graph and is designed to evaluate both the accuracy and completeness of dense captions at a granular level. Additionally, we identify the root cause of hallucination as the model's over-reliance on its language prior. To address this, we propose PerturboLLaVA, which reduces the model's reliance on the language prior by incorporating adversarially perturbed text during training. This method enhances the model's focus on visual inputs, effectively reducing hallucinations and producing accurate, image-grounded descriptions without incurring additional computational overhead. PerturboLLaVA significantly improves the fidelity of generated captions, outperforming existing approaches in handling multimodal hallucinations and achieving improved performance across general multimodal benchmarks.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 44 references that are in the map. First posted on arXiv 9 Mar 2025.

Builds on

34 of 44 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  2. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  3. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  4. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  5. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models23 Oct
  6. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  7. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  8. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 26

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  12. 2024Qwen2 Technical Reportcited by 2,120
14 more
  1. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  2. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  3. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  4. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  5. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  6. 2018Object Hallucination in Image Captioningcited by 669
  7. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  8. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  9. 2019Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biasescited by 280
  10. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  11. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 169
  12. 2016Visual Question Answering: A Survey of Methods and Datasetscited by 123
  13. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 108
  14. 2023VIGC: Visual Instruction Generation and Correctioncited by 80

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 19; 2026: 19.

38 papers.

Sort:

By area:

  1. 2025Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMscited by 55
  2. 2025MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Modelscited by 12
  3. 2026On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMscited by 11
  4. 2025OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucinationcited by 9
  5. 2025Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attentioncited by 9
  6. 2025Cross-Image Contrastive Decoding: Precise, Lossless Suppression of Language Priors in Large Vision-Language Modelscited by 8
  7. 2025JEEM: Vision-Language Understanding in Four Arabic Dialectscited by 7
  8. 2025Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigationcited by 7
  9. 2026MemDreamer: Decoupling Perception and Reasoning for Long Video Understanding via Hierarchical Graph Memory and Agentic Retrieval Mechanismcited by 7
  10. 2026MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimizationcited by 6
  11. 2026Android Coach: Improve Online Agentic Training Efficiency with Single State Multiple Actionscited by 6
  12. 2025Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigationcited by 5
26 more
  1. 2025InEx: Hallucination Mitigation via Introspection and Cross-Modal Multi-Agent Collaborationcited by 5
  2. 2026GraphThinker: Reinforcing Temporally Grounded Video Reasoning with Event Graph Thinkingcited by 5
  3. 2025Taming Hallucinations: Boosting MLLMs' Video Understanding via Counterfactual Video Generationcited by 4
  4. 2026Do MLLMs Really See It: Reinforcing Visual Attention in Multimodal LLMscited by 4
  5. 2026Revealing and Enhancing Core Visual Regions: Harnessing Internal Attention Dynamics for Hallucination Mitigation in LVLMscited by 4
  6. 2026FINER: MLLMs Hallucinate under Fine-grained Negative Queriescited by 4
  7. 2026FREAK: A Fine-grained Hallucination Evaluation Benchmark for Advanced MLLMscited by 4
  8. 2025Grounding or Guessing? Visual Signals for Detecting Hallucinations in Sign Language Translationcited by 3
  9. 2025Towards Fine-Grained Human Motion Video Captioningcited by 3
  10. 2025Mitigating Object and Action Hallucinations in Multimodal LLMs via Self-Augmented Contrastive Alignmentcited by 3
  11. 2026Evidence-RL: Towards Evidence-intensive Visual Reasoningcited by 3
  12. 2025Investigating and Enhancing the Robustness of Large Multimodal Models Against Temporal Inconsistencycited by 2
  13. 2025Anatomical Region-Guided Contrastive Decoding: A Plug-and-Play Strategy for Mitigating Hallucinations in Medical VLMscited by 2
  14. 2025Watch Closely: Mitigating Object Hallucinations in Large Vision-Language Models with Disentangled Decodingcited by 2
  15. 2026TGIF: Text-Guided Layer Fusion Mitigates Hallucination in Multimodal LLMscited by 2
  16. 2025Curing Semantic Drift: A Dynamic Approach to Grounding Generation in Large Vision-Language Modelscited by 1
  17. 2026HII-DPO: Eliminate Hallucination via Accurate Hallucination-Inducing Counterfactual Imagescited by 1
  18. 2026Are Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial Environmentscited by 1
  19. 2025Rationale-Enhanced Decoding for Multi-modal Chain-of-Thoughtcited by 0
  20. 2025TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMscited by 0
  21. 2026Schr"oMind: Mitigating Hallucinations in Multimodal Large Language Models via Solving the Schr"odinger Bridge Problemcited by 0
  22. 2026Scalpel: Fine-Grained Alignment of Attention Activation Manifolds via Mixture Gaussian Bridges to Mitigate Multimodal Hallucinationcited by 0
  23. 2026Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguardscited by 0
  24. 2026Beyond Language Priors: Diagnosing and Fixing Visual-Origin Hallucinations in Multimodal LLMcited by 0
  25. 2026MLLMs Hallucinate when Information Distribution Drifts in Synergy Headscited by 0
  26. 2026PerturBot: Breaking Shortcut Priors in Vision-Language-Action Models with Perturbative Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024NoiseBoost: Alleviating Hallucination with Noise Perturbation for Multimodal Large Language Models
  2. 2025Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimization
  3. 2024Hallu-PI: Evaluating Hallucination in Multi-modal Large Language Models within Perturbed Inputs
  4. 2023HallE-Control: Controlling Object Hallucination in Large Multimodal Models
  5. 2024Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decoding
  6. 2024Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage
  7. 2025Low-hallucination Synthetic Captions for Large-Scale Vision-Language Model Pre-training
  8. 2024Skip n: A Simple Method to Reduce Hallucination in Large Vision-Language Models