mlsanityData to 7 Oct 2026

Order Matters: Exploring Order Sensitivity in Multimodal Large Language Models

arXiv 2410.16983 ↗First posted 22 Oct 2024cs.AIcited by 19

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) utilize multimodal contexts consisting of text, images, or videos to solve various multimodal tasks. However, we find that changing the order of multimodal input can cause the model's performance to fluctuate between advanced performance and random guessing. This phenomenon exists in both single-modality (text-only or image-only) and mixed-modality (image-text-pair) contexts. Furthermore, we demonstrate that popular MLLMs pay special attention to certain multimodal context positions, particularly the beginning and end. Leveraging this special attention, we place key video frames and important image/text content in special positions within the context and submit them to the MLLM for inference. This method results in average performance gains of 14.7% for video-caption matching and 17.8% for visual question answering tasks. Additionally, we propose a new metric, Position-Invariant Accuracy (PIA), to address order bias in MLLM evaluation. Our research findings contribute to a better understanding of Multi-Modal In-Context Learning (MMICL) and provide practical strategies for enhancing MLLM performance without increasing computational costs.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 51 references that are in the map. First posted on arXiv 22 Oct 2024.

Builds on

32 of 51 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark7 Feb
  2. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model29 Jan
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  4. 2023Generative Multimodal Models are In-Context Learners20 Dec

Other references · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  5. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  6. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  7. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  8. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  9. 2019How Can We Know What Language Models Know?cited by 953
  10. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  11. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  12. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
16 more
  1. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  2. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  3. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  4. 2021GPT3Mix: Leveraging Large-scale Language Models for Text Augmentationcited by 185
  5. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 170
  6. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 160
  7. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
  8. 2023On the Adversarial Robustness of Multi-Modal Foundation Modelscited by 142
  9. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMscited by 79
  10. 2023Split and Merge: Aligning Position Biases in LLM-based Evaluatorscited by 71
  11. 2022High-fidelity 3D GAN Inversion by Pseudo-multi-view Optimizationcited by 65
  12. 2023Exploring Diverse In-Context Configurations for Image Captioningcited by 63
  13. 2021Reordering Examples Helps during Priming-based Few-Shot Learningcited by 50
  14. 2024Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMscited by 30
  15. 2024Batch-ICL: Effective, Efficient, and Order-Agnostic In-Context Learningcited by 20
  16. 2024Hard Prompts Made Interpretable: Sparse Entropy Regularization for Prompt Tuning with RLcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 12.

19 papers.

Sort:

By area:

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  2. 2025Who is in the Spotlight: The Hidden Bias Undermining Multimodal Retrieval-Augmented Generationcited by 9
  3. 2026Residual Decoding: Mitigating Hallucinations in Large Vision-Language Models via History-Aware Residual Guidancecited by 5
  4. 2025GraphSOS: Graph Sampling and Order Selection to Help LLMs Understand Graphs Bettercited by 4
  5. 2026ViKey: Enhancing Temporal Understanding in Videos via Visual Promptingcited by 3
  6. 2025MLLM-based Speech Recognition: When and How is Multimodality Beneficial?cited by 1
  7. 2025OMHBench: Benchmarking Balanced and Grounded Omni-Modal Multi-Hop Reasoningcited by 1
  8. 2025Unexplored flaws in multiple-choice VQA make benchmarking unreliablecited by 1
  9. 2026MetaRA: Metamorphic Robustness Assessment for Multimodal Large Language Model-based Visual Question Answering Systemscited by 1
  10. 2026Detecting Differences Is Not Understanding Structure: Large Language Models Fail at Graph Isomorphismcited by 1
  11. 2026Lost at the End: Primacy Bias in Multimodal Retrieval-Augmented Question Answeringcited by 1
  12. 2025Mitigating Hallucinations on Object Attributes using Multiview Images and Negative Instructionscited by 0
7 more
  1. 2026Where is the multimodal goal post? On the Ability of Foundation Models to Recognize Contextually Important Momentscited by 0
  2. 2026Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMscited by 0
  3. 2026Where Do Vision-Language Models Fail? World Scale Analysis for Image Geolocalizationcited by 0
  4. 2026Logit-Attention Divergence: Mitigating Position Bias in Multi-Image Retrieval via Attention-Guided Calibrationcited by 0
  5. 2026Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Modelscited by 0
  6. 2026Ask Twice, Look Twice: Prompt Echoing Resolves the Question-First Paradox in Vision-Language Modelscited by 0
  7. 2026Same evidence, different judgments: Evidence noncommutative in vision/speech-text conflictscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Burn After Reading: Do Multimodal Large Language Models Truly Capture Order of Events in Image Sequences?
  2. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  3. 2024A Survey on Benchmarks of Multimodal Large Language Models
  4. 2024II-Bench: An Image Implication Understanding Benchmark for Multimodal Large Language Models
  5. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models
  6. 2025When Language Overrules: Revealing Text Dominance in Multimodal Large Language Models
  7. 2025How Multimodal LLMs Solve Image Tasks: A Lens on Visual Grounding, Task Reasoning, and Answer Decoding
  8. 2024Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Models