mlsanityData to 7 Oct 2026

Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projection

arXiv 2412.13817 ↗First posted 18 Dec 2024cs.CVcited by 46

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Recent studies have shown that large vision-language models (LVLMs) often suffer from the issue of object hallucinations (OH). To mitigate this issue, we introduce an efficient method that edits the model weights based on an unsafe subspace, which we call HalluSpace in this paper. With truthful and hallucinated text prompts accompanying the visual content as inputs, the HalluSpace can be identified by extracting the hallucinated embedding features and removing the truthful representations in LVLMs. By orthogonalizing the model weights, input features will be projected into the Null space of the HalluSpace to reduce OH, based on which we name our method Nullu. We reveal that HalluSpaces generally contain prior information in the large language models (LLMs) applied to build LVLMs, which have been shown as essential causes of OH in previous studies. Therefore, null space projection suppresses the LLMs' priors to filter out the hallucinated features, resulting in contextually accurate outputs. Experiments show that our method can effectively mitigate OH across different LVLM families without extra inference costs and also show strong performance in general LVLM benchmarks. Code is released at https://github.com/Ziwei-Zheng/Nullu.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 59 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

49 of 59 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  2. 2024Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Models24 Jun
  3. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  4. 2024Multi-Modal Hallucination Control by Visual Information Grounding20 Mar
  5. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  6. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding28 Feb
  7. 2024Visual Hallucinations of Multi-modal Large Language Models22 Feb
  8. 2024Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Models18 Feb
  9. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Model12 Dec
  10. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  11. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  12. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
4 more
  1. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  2. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  3. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  4. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2023Qwen Technical Reportcited by 3,273
21 more
  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  5. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  6. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  7. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  8. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  9. 2018Object Hallucination in Image Captioningcited by 661
  10. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  11. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  12. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  13. 2017Beam Search Strategies for Neural Machine Translationcited by 232
  14. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 99
  15. 2021Let there be a clock on the beach: Reducing Object Hallucination in Image Captioningcited by 74
  16. 2023Exposing and Mitigating Spurious Correlations for Cross-Modal Retrievalcited by 35
  17. 2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMscited by 35
  18. 2024Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicitycited by 33
  19. 2024FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedbackcited by 30
  20. 2024Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editingcited by 23
  21. 2024Can LVLMs Obtain a Driver's License? A Benchmark Towards Reliable AGI for Autonomous Drivingcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 11; 2026: 35.

46 papers.

Sort:

By area:

  1. 2025Steer LLM Latents for Hallucination Detectioncited by 49
  2. 2025TruthPrInt: Mitigating Large Vision-Language Models Object Hallucination Via Latent Truthful-Guided Pre-Interventioncited by 11
  3. 2025When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Modelscited by 11
  4. 2026Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steeringcited by 8
  5. 2025Short-LVLM: Compressing and Accelerating Large Vision-Language Models by Pruning Redundant Layerscited by 7
  6. 2026Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigationcited by 7
  7. 2025GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMscited by 6
  8. 2026VAUQ: Vision-Aware Uncertainty Quantification for LVLM Self-Evaluationcited by 5
  9. 2026Fighting Hallucinations with Counterfactuals: Diffusion-Guided Perturbations for LVLM Hallucination Suppressioncited by 5
  10. 2025Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinationscited by 3
  11. 2025Tracing and Mitigating Hallucinations in Multimodal LLMs via Dynamic Attention Localizationcited by 3
  12. 2025"It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Modelscited by 3
34 more
  1. 2026Revis: Sparse Latent Steering to Mitigate Object Hallucination in Large Vision-Language Modelscited by 3
  2. 2026Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbationcited by 3
  3. 2026Mitigating Hallucinations in Large Vision-Language Models without Performance Degradationcited by 3
  4. 2026Object Hallucination-Free Reinforcement Unlearning for Vision-Language Modelscited by 3
  5. 2026MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Modelscited by 2
  6. 2026HulluEdit: Single-Pass Evidence-Consistent Subspace Editing for Mitigating Hallucinations in Large Vision-Language Modelscited by 2
  7. 2026Locate-then-Sparsify: Attribution Guided Sparse Strategy for Visual Hallucination Mitigationcited by 2
  8. 2025Suppressing VLM Hallucinations with Spectral Representation Filteringcited by 1
  9. 2026Context-Aware Decoding for Faithful Vision-Language Generationcited by 1
  10. 2026HIME: Mitigating Object Hallucinations in LVLMs via Hallucination Insensitivity Model Editingcited by 1
  11. 2026Risk-aware Selective Prompting for Hallucination Mitigation in Large Vision-Language Modelscited by 1
  12. 2026Safe Autoregressive Image Generation with Iterative Self-Improving Codebookscited by 1
  13. 2025One SPACE to Rule Them All: Jointly Mitigating Factuality and Faithfulness Hallucinations in LLMscited by 0
  14. 2025Look Closer! An Adversarial Parametric Editing Framework for Hallucination Mitigation in VLMscited by 0
  15. 2026Countering the Over-Reliance Trap: Mitigating Object Hallucination for LVLMs via a Self-Validation Frameworkcited by 0
  16. 2026Two Birds, One Projection: Harmonizing Safety and Utility in LVLMs via Inference-time Feature Projectioncited by 0
  17. 2026Hallucination-aware intermediate representation edit in large vision-language modelscited by 0
  18. 2026Mitigating Entangled Steering in Large Vision-Language Models for Hallucination Reductioncited by 0
  19. 2026Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibrationcited by 0
  20. 2026What Makes LVLMs Hallucinate Less? Unveiling the Architectural Factors Behind Hallucination Robustnesscited by 0
  21. 2026Multi-FRuGaL: Multimodal Flexible Redundancy-aware Decomposed Gated Learning for Cancer Diagnosis and Prognosiscited by 0
  22. 2026Spectral Query-Key Product Weight Steering for Training-Free VLM Hallucination Mitigationcited by 0
  23. 2026Look Clearly Before Answering: Mitigating Hallucinations in LVLMs via Saliency-Driven Perceptual Realignmentcited by 0
  24. 2026HalluScope: Fine-grained Hallucination Diagnosis for Multimodal Large Language Modelscited by 0
  25. 2026Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Modelscited by 0
  26. 2026Prior Directions: Why GUI Grounding Gets Locked in the Pastcited by 0
  27. 2026Gradient Immunity: Null-Space Resistance to Malicious Fine-Tuningcited by 0
  28. 2026Wiener Representation Filtering for VLM Hallucination Suppressioncited by 0
  29. 2026Beyond Global Editing: Per-Instance Disentangled Subspaces for Training-Free Hallucination Mitigation in LVLMscited by 0
  30. 2026Test-Time Hallucination Control in Large Vision-Language Modelscited by 0
  31. 2026EviAnchor: Mitigating Hallucinations in Large Vision-Language Models via Regional Visual Evidence Compensationcited by 0
  32. 2026Reducing Hallucinated Transcripts in Whisper via Hallucination Space Projectioncited by 0
  33. 2026When Helpful Text Hurts: Option-Redirecting Bias in Vision-Language Modelscited by 0
  34. 2026DARE to Mitigate Hallucination: Dual-path Auto-Regressive-aware Editingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decoding
  2. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering
  3. 2026NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors
  4. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations
  5. 2024From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models
  6. 2025Watch Closely: Mitigating Object Hallucinations in Large Vision-Language Models with Disentangled Decoding
  7. 2025Evaluating Hallucination in Large Vision-Language Models based on Context-Aware Object Similarities
  8. 2025A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models