# Skip n: A Simple Method to Reduce Hallucination in Large Vision-Language Models

arXiv: https://arxiv.org/abs/2402.01345 · HTML: https://mlsanity.com/paper/2402.01345/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 25 of its 34 references that are in the map.
- First posted on arXiv 2 Feb 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Hallucination mitigation in LVLMs](https://mlsanity.com/mllm/hallucination-mitigation-in-lvlms/)

## Builds on (most cited first, 25 of 25)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [Aligning Large Multimodal Models with Factually Augmented RLHF](https://mlsanity.com/paper/2309.14525/) (2023)
- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [Object Hallucination in Image Captioning](https://mlsanity.com/paper/1809.02156/) (2018)
- [MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning](https://mlsanity.com/paper/2310.09478/) (2023)
- [OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation](https://mlsanity.com/paper/2311.17911/) (2023)
- [RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback](https://mlsanity.com/paper/2312.00849/) (2023)
- [DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models](https://mlsanity.com/paper/2309.03883/) (2023)
- [Analyzing and Mitigating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2310.00754/) (2023)
- [Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization](https://mlsanity.com/paper/2311.16839/) (2023)
- [A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity](https://mlsanity.com/paper/2305.13169/) (2023)
- [Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challenges](https://mlsanity.com/paper/2311.03287/) (2023)
- [What's In My Big Data?](https://mlsanity.com/paper/2310.20707/) (2023)
- [Evaluation and Analysis of Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2308.15126/) (2023)
- [VIGC: Visual Instruction Generation and Correction](https://mlsanity.com/paper/2308.12714/) (2023)
- [UMIX: Improving Importance Weighting for Subpopulation Shift via Uncertainty-Aware Mixup](https://mlsanity.com/paper/2209.08928/) (2022)
- [Parrot Captions Teach CLIP to Spot Text](https://mlsanity.com/paper/2312.14232/) (2023)
- [Reweighted Mixup for Subpopulation Shift](https://mlsanity.com/paper/2304.04148/) (2023)

## Built on by (most cited first, 9 of 9)

- [RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthiness](https://mlsanity.com/paper/2405.17220/) (2024)
- [Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key](https://mlsanity.com/paper/2501.09695/) (2025)
- [Mitigating Object Hallucinations via Sentence-Level Early Intervention](https://mlsanity.com/paper/2507.12455/) (2025)
- [Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)](https://mlsanity.com/paper/2504.03151/) (2025)
- [Unified Triplet-Level Hallucination Evaluation for Large Vision-Language Models](https://mlsanity.com/paper/2410.23114/) (2024)
- [Mitigating Object Hallucinations in MLLMs via Multi-Frequency Perturbations](https://mlsanity.com/paper/2503.14895/) (2025)
- [MARLIN: Multi-Agent Reinforcement Learning Guided by Language-Based Inter-Robot Negotiation](https://mlsanity.com/paper/2410.14383/) (2024)
- [Reflective Preference Optimization (RPO): Enhancing On-Policy Alignment via Hint-Guided Reflection](https://mlsanity.com/paper/2512.13240/) (2025)
- [Causal Decoding for Hallucination-Resistant Multimodal Large Language Models](https://mlsanity.com/paper/2602.21441/) (2026)

## Abstract (arXiv)

Recent advancements in large vision-language models (LVLMs) have demonstrated impressive capability in visual information understanding with human language. Despite these advances, LVLMs still face challenges with multimodal hallucination, such as generating text descriptions of objects that are not present in the visual information. However, the underlying fundamental reasons of multimodal hallucinations remain poorly explored. In this paper, we propose a new perspective, suggesting that the inherent biases in LVLMs might be a key factor in hallucinations. Specifically, we systematically identify a semantic shift bias related to paragraph breaks (nn), where the content before and after 'nn' in the training data frequently exhibit significant semantic changes. This pattern leads the model to infer that the contents following 'nn' should be obviously different from the preceding contents with less hallucinatory descriptions, thereby increasing the probability of hallucinatory descriptions subsequent to the 'nn'. We have validated this hypothesis on multiple publicly available LVLMs. Besides, we find that deliberately inserting 'nn' at the generated description can induce more hallucinations. A simple method is proposed to effectively mitigate the hallucination of LVLMs by skipping the output of 'n'.