mlsanityData to 14 Oct 2026

Logit-Attention Divergence: Mitigating Position Bias in Multi-Image Retrieval via Attention-Guided Calibration

arXiv 2605.11591 ↗First posted 12 May 2026cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have shown strong performance in multi-image cross-modal retrieval, yet suffer from severe position bias, where predictions are dominated by input order rather than semantic relevance. Through empirical analysis, we identify a phenomenon termed Logit-Attention Divergence, in which output logits are heavily biased while internal attention maps remain well-aligned with relevant visual evidence. This observation reveals a fundamental limitation of existing logit-level calibration methods such as PriDe. Based on this insight, we propose a training-free, attention-guided debiasing framework that leverages intrinsic attention signals for instance-level correction at inference time, requiring only a minimal calibration set with negligible computational overhead. Experiments on MS-COCO-based benchmarks show that our method substantially improves permutation invariance and achieves state-of-the-art performance, enhancing accuracy by over 40% compared to baselines. Code is available at https://github.com/brightXian/LAD.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 35 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

30 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Generative Cross-Modal Retrieval: Memorizing Images in Multimodal Language Models for Retrieval and Beyond16 Feb

Other references · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  9. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  12. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
17 more
  1. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  2. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  3. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  4. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 256
  5. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 244
  6. 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 90
  7. 2022Parallel Context Windows for Large Language Modelscited by 81
  8. 2024"My Answer is C": First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Modelscited by 69
  9. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Modelscited by 67
  10. 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 65
  11. 2025On the Emergence of Position Bias in Transformerscited by 59
  12. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Modelscited by 55
  13. 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Imagescited by 39
  14. 2025Identifying and Mitigating Position Bias of Multi-image Vision-Language Modelscited by 25
  15. 2024MMREC: LLM Based Multi-Modal Recommender Systemcited by 22
  16. 2024Order Matters: Exploring Order Sensitivity in Multimodal Large Language Modelscited by 20
  17. 2025Embedding-based Retrieval in Multimodal Content Moderationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026The Evidence Anchoring Boundary: Where Visual Interventions Matter and When They Help in Multimodal LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Attention Debiasing for Token Pruning in Vision Language Models
  2. 2024Debiasing Multimodal Large Language Models via Penalization of Language Priors
  3. 2026Can Retrieval Heads See Images? Multimodal Retrieval Heads in Long-Context Vision-Language Models
  4. 2024Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models
  5. 2026Benchmarking Deflection and Hallucination in Large Vision-Language Models
  6. 2026Indexing Multimodal Language Models for Large-scale Image Retrieval
  7. 2025Who is in the Spotlight: The Hidden Bias Undermining Multimodal Retrieval-Augmented Generation
  8. 2026The Cost of Context: Mitigating Textual Bias in Multimodal Retrieval-Augmented Generation