mlsanityData to 7 Oct 2026

Identifying and Mitigating Position Bias of Multi-image Vision-Language Models

arXiv 2503.13792 ↗First posted 18 Mar 2025cs.CVcited by 24

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

The evolution of Large Vision-Language Models (LVLMs) has progressed from single to multi-image reasoning. Despite this advancement, our findings indicate that LVLMs struggle to robustly utilize information across multiple images, with predictions significantly affected by the alteration of image positions. To further explore this issue, we introduce Position-wise Question Answering (PQA), a meticulously designed task to quantify reasoning capabilities at each position. Our analysis reveals a pronounced position bias in LVLMs: open-source models excel in reasoning with images positioned later but underperform with those in the middle or at the beginning, while proprietary models show improved comprehension for images at the beginning and end but struggle with those in the middle. Motivated by this, we propose SoFt Attention (SoFA), a simple, training-free approach that mitigates this bias by employing linear interpolation between inter-image causal attention and bidirectional counterparts. Experimental results demonstrate that SoFA reduces position bias and enhances the reasoning performance of existing LVLMs.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 60 of its 74 references that are in the map. First posted on arXiv 18 Mar 2025.

Builds on

60 of 74 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Building and better understanding vision-language models: insights and future directions22 Aug
  2. 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning18 Jun
  3. 2024What matters when building vision-language models?3 May
  4. 2023How to Configure Good In-Context Sequence for Visual Question Answering4 Dec
  5. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models2 Aug
  6. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 53

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  11. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  12. 2023Qwen Technical Reportcited by 3,273
38 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  8. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  9. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  10. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  11. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  12. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  13. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  14. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  15. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  16. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  17. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  18. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  19. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  20. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  21. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  22. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  23. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 273
  24. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 212
  25. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  26. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 178
  27. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  28. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  29. 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 89
  30. 2022Parallel Context Windows for Large Language Modelscited by 81
  31. 2024Many-Shot In-Context Learning in Multimodal Foundation Modelscited by 64
  32. 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 61
  33. 2023LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matchingcited by 60
  34. 2023Large Language Models are Visual Reasoning Coordinatorscited by 58
  35. 2024Needle In A Multimodal Haystackcited by 56
  36. 2024What Makes Multimodal In-Context Learning Work?cited by 54
  37. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Modelscited by 53
  38. 2024CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMscited by 49

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 11; 2026: 13.

24 papers.

Sort:

By area:

  1. 2025More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Modelscited by 26
  2. 2025PeRL: Permutation-Enhanced Reinforcement Learning for Interleaved Vision-Language Reasoningcited by 11
  3. 2025PixelCraft: A Multi-Agent System for High-Fidelity Visual Reasoning on Structured Imagescited by 11
  4. 2025Unlocking Vision-Language Models for Video Anomaly Detection via Fine-Grained Promptingcited by 8
  5. 2025TRAP: Targeted Redirecting of Agentic Preferencescited by 4
  6. 2025Hearing the Order: Investigating Position Bias in Large Audio-Language Modelscited by 4
  7. 2025Training Multi-Image Vision Agents via End2End Reinforcement Learningcited by 4
  8. 2026Subliminal Signals in Preference Labelscited by 3
  9. 2025Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMscited by 2
  10. 2025Beyond the Vision Encoder: Identifying and Mitigating Spatial Bias in Large Vision-Language Modelscited by 2
  11. 2025Prune-Then-Plan: Step-Level Calibration for Stable Frontier Exploration in Embodied Question Answeringcited by 2
  12. 2026Looking Back and Forth: Cross-Image Attention Calibration and Attentive Preference Learning for Multi-Image Hallucination Mitigationcited by 2
12 more
  1. 2026SLVMEval: Synthetic Meta Evaluation Benchmark for Text-to-Long Video Generationcited by 2
  2. 2026Decouple and Cache: KV Cache Construction for Streaming Video Understandingcited by 2
  3. 2026DynaTok: Temporally Adaptive and Positional Bias-Aware Token Compression for Video-LLMscited by 2
  4. 2025The Describe-Then-Generate Bottleneck: How VLM Descriptions Alter Image Generation Outcomescited by 1
  5. 2026Order Is Not Layout: Order-to-Space Bias in Image Generationcited by 1
  6. 2026ASAP: Attention-Shift-Aware Pruning for Efficient LVLM Inferencecited by 1
  7. 2026SSMNBench: Diagnosing Image-based Cross-View Human-Object Understanding via Single-View Sufficiency and Multi-View Necessitycited by 1
  8. 2026Break the Brake, Not the Wheel: Untargeted Jailbreak via Entropy Maximizationcited by 0
  9. 2026Logit-Attention Divergence: Mitigating Position Bias in Multi-Image Retrieval via Attention-Guided Calibrationcited by 0
  10. 2026TASTE: A Designer-Annotated Multi-Dimensional Preference Dataset for AI-Generated Graphic Designcited by 0
  11. 2026C3-Bench: A Context-Aware Change Captioning Benchmarkcited by 0
  12. 2026Uncovering Ordinal-Matching Bias in Audio-Visual LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024FiVL: A Framework for Improved Vision-Language Alignment through the Lens of Training, Evaluation and Explainability
  2. 2025MPCAR: Multi-Perspective Contextual Augmentation for Enhanced Visual Reasoning in Large Vision-Language Models
  3. 2024Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance
  4. 2025Benchmarking and Mitigating MCQA Selection Bias of Large Vision-Language Models
  5. 2025When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs
  6. 2025Elevating Visual Question Answering through Implicitly Learned Reasoning Pathways in LVLMs
  7. 2026From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models
  8. 2023Probing the Role of Positional Information in Vision-Language Models