Dual-Stream Cross-Anchor Correction Grounding Long-Form Captions and the Domain Limits of Object-Level Anchors
On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs
Abstract (arXiv)
Object hallucination in multimodal large language models arises when language priors and corpus co-occurrence bias outweigh the visual evidence, with nothing tying an object mention to the image. Most remedies intervene at decoding time, yet under a unified protocol their benefit is confined to short captions; supervised fine-tuning (SFT) on a detail-rich corpus lengthens captions, but over forty percent still name absent objects. This paper proposes Dual-Stream Cross-Anchor Correction (DSCC). Unlike work that post-processes decoding, DSCC injects object-level visual anchors into the language model itself during fine-tuning: a perception stream aligns object-level hidden states at an intermediate layer to frozen text anchors by a bidirectional contrastive objective; a cognition stream lets deeper layers query those anchors by cross-attention at every generation step; and a two-stage curriculum gate couples them, making evidence retrieval a structural constraint on generation. Under one backbone and one scoring protocol, experiments span long-caption hallucination, object-existence discrimination and cross-domain generalisation, with vanilla SFT on the same corpus and schedule as a length- and density-matched control separating the data effect from the architectural gain. DSCC alone reaches the long-caption, low-hallucination region: captions roughly 1.9 times the baseline length at 88.19% precision per object mention, the highest under a density-independent criterion. Ablations expose a synergy: the perception stream alone degrades precision yet reverses sign when stacked on the cognition stream. No universal superiority is claimed: three out-of-domain benchmarks yield a predictable, falsifiable domain-conditionality, the synergy being bound to the anchors' semantic domain and breaking on charts and illusions.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 46 references that are in the map. First posted on arXiv 13 Aug 2026.
Builds on
43 of 46 references.
Closest ancestors · 16
Same direction, newest first
- 2025Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context23 Oct
- 2024Mitigating Object Hallucination via Concentric Causal Attention21 Oct
- 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models4 Aug
- 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
- 2024Multi-Modal Hallucination Control by Visual Information Grounding20 Mar
- 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
- 2024Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Models18 Feb
- 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
- 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
- 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
- 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data22 Nov
- 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision13 Nov
4 more
- 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
- 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models23 Oct
- 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
- 2023Evaluating Object Hallucination in Large Vision-Language Models17 May
Other references · 27
- 2017Attention Is All You Needcited by 59,932
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2023Visual Instruction Tuningcited by 7,792
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
15 more
- 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
- 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
- 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
- 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
- 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
- 2021Grounded Language-Image Pre-trainingcited by 1,147
- 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
- 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
- 2018Object Hallucination in Image Captioningcited by 661
- 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
- 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
- 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
- 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
- 2024Calibrated Self-Rewarding Vision Language Modelscited by 77
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Does Object Grounding Really Reduce Hallucination of Large Vision-Language Models?
- 2024CrossCheckGPT: Universal Hallucination Ranking for Multimodal Foundation Models
- 2026Visual Attention Drifts,but Anchors Hold:Mitigating Hallucination in Multimodal Large Language Models via Cross-Layer Visual Anchors
- 2026Causal Decoding for Hallucination-Resistant Multimodal Large Language Models
- 2026Targeting the Attention Heads Behind Object Hallucination in LLaVA
- 2026Rethinking CD: A Reproducibility Study and Extension on the Ineffectiveness of Contrastive Decoding at Mitigating Object Hallucinations in MLLMs
- 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites
- 2024Do More Details Always Introduce More Hallucinations in LVLM-based Image Captioning?