mlsanityData to 14 Oct 2026

Looking Back and Forth: Cross-Image Attention Calibration and Attentive Preference Learning for Multi-Image Hallucination Mitigation

arXiv 2603.07048 ↗First posted 7 Mar 2026cs.CV cs.AIcited by 2

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

also in 1 more
Abstract (arXiv)

Although large vision-language models (LVLMs) have demonstrated remarkable capabilities, they are prone to hallucinations in multi-image tasks. We attribute this issue to limitations in existing attention mechanisms and insufficient cross-image modeling. Inspired by this, we propose a structured hallucination mitigation framework involving Cross-Image Attention calibration and Preference Learning (CAPL). CAPL explicitly enhances inter-image interactions at the architectural level while reinforcing reliance on genuine cross-image evidence during training, thereby improving the model's perception and modeling of cross-image associations. Specifically, we (i) introduce a selectable image token interaction attention mechanism to establish fine-grained cross-image entity alignment and information flow; (ii) design a cross-image modeling-based preference optimization strategy that contrasts reasoning outcomes under full inter-image interaction and those obtained when images are mutually invisible, encouraging the model to ground its predictions in authentic visual evidence and mitigating erroneous inferences driven by textual priors. Experimental results demonstrate that CAPL consistently improves performance across multiple model architectures, achieving stable gains on both multi-image hallucination and general benchmarks. Notably, performance on single-image visual tasks remains stable or slightly improves, indicating strong generalization capability.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 50 references that are in the map. First posted on arXiv 7 Mar 2026.

Builds on

45 of 50 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Hallucination at a Glance: Controlled Visual Edits and Fine-Grained Multimodal Learning8 Jun
  2. 2025Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs26 May
  3. 2024Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMs24 May
  4. 2024Multi-Modal Hallucination Control by Visual Information Grounding20 Mar
  5. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb
  6. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Model12 Dec
  7. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  8. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  9. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  10. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  11. 2023CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuning5 Sep
  12. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
1 more
  1. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training14 Oct

Other references · 32

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2025Qwen3-VL Technical Reportcited by 2,793
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  8. 2015VQA: Visual Question Answeringcited by 2,506
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  10. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  11. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  12. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
20 more
  1. 2018Object Hallucination in Image Captioningcited by 669
  2. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  3. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  4. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  5. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 355
  6. 2024What matters when building vision-language models?cited by 326
  7. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
  8. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 216
  9. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 188
  10. 2021Causal Attention for Vision-Language Taskscited by 111
  11. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 101
  12. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMscited by 90
  13. 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoningcited by 54
  14. 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Imagescited by 39
  15. 2024MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Modelscited by 34
  16. 2025VLM2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit Matching Visual Cuescited by 33
  17. 2025Identifying and Mitigating Position Bias of Multi-image Vision-Language Modelscited by 25
  18. 2025PeRL: Permutation-Enhanced Reinforcement Learning for Interleaved Vision-Language Reasoningcited by 11
  19. 2026Seeing Through the Chain: Mitigate Hallucination in Multimodal Reasoning Models via CoT Compression and Contrastive Preference Optimizationcited by 5
  20. 2025Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Reasoning Matters: Mitigate Hallucination in Multimodal Large Reasoning Models via Reasoning-Conditioned Preference Optimizationcited by 2
  2. 2026MergeHEIR: Mitigating Multimodal Hallucinations as the Tax of Model Mergingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMs
  2. 2026Rethinking Visual Neglect: Steering via Context-Preference for MLLM Hallucination Mitigation
  3. 2026Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models
  4. 2025Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs
  5. 2025MIHBench: Benchmarking and Mitigating Multi-Image Hallucinations in Multimodal Large Language Models
  6. 2026Fine-Grained Multi Image Object Hallucination Benchmark
  7. 2025Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation
  8. 2026Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigation