mlsanityData to 7 Oct 2026

Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models

arXiv 2606.26079 ↗First posted 24 Jun 2026cs.CL cs.CV cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Standard benchmarks for multimodal large language models (MLLMs) score each item on one canonical ordering and miss whether order-irrelevant shuffling changes the answer, a baseline reliability property called for by emerging AI evaluation guidelines. We introduce Facet-Probe, a five-facet audit (option, evidence-chunk, document-rank, image-set, and mixed-modality ordering) of 18 frontier and open-weight MLLMs. A Bayesian item-response model separates ordering noise from per-facet bias, and a same-ordering control estimates the decoder-stochastic floor for observed flips. We find that none of the 18 MLLMs we audit are order-invariant: screened per-facet panel-mean flip rates span 24-50%. A Gemini same-ordering control at temperature 0 estimates a substantial ordering excess over a same-input decoder-noise floor in verified cells. Capability predicts but does not eliminate flips; the best model still flips on 13.4% of trials. In our Gemini mitigation tests, training-free prompt changes are modality-conditional and do not transfer from text to visual reasoning. These results suggest that prompt-level mitigation alone is unlikely to provide general order robustness, motivating future work on training-time and architectural approaches. We propose cross-ordering flip rate as a standard reporting axis for MLLMs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 69 references that are in the map. First posted on arXiv 24 Jun 2026.

Builds on

31 of 69 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theory3 Mar
  2. 2025Promptception: How Sensitive Are Large Multimodal Models to Prompts?4 Sep
  3. 2024Order Matters: Exploring Order Sensitivity in Multimodal Large Language Models22 Oct
  4. 2024MANTIS: Interleaved Multi-Image Instruction Tuning2 May
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 26

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  3. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  4. 2025OpenAI GPT-5 System Cardcited by 910
  5. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  6. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  7. 2025MedGemma Technical Reportcited by 398
  8. 2025Kimi-VL Technical Reportcited by 381
  9. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 295
  10. 2025MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understandingcited by 212
  11. 2019Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyrocited by 187
  12. 2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 177
14 more
  1. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 139
  2. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  3. 2024Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilizationcited by 97
  4. 2023Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experimentcited by 71
  5. 2024Premise Order Matters in Reasoning with Large Language Modelscited by 71
  6. 2025Measurement to Meaning: A Validity-Centered Framework for AI Evaluationcited by 54
  7. 2025Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answeringcited by 33
  8. 2025MedFrameQA: A Multi-Image Medical VQA Benchmark for Clinical Reasoningcited by 33
  9. 2023Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Modelscited by 30
  10. 2026Brittlebench: Quantifying LLM robustness via prompt sensitivitycited by 19
  11. 2025MRAMG-Bench: A Comprehensive Benchmark for Advancing Multimodal Retrieval-Augmented Multimodal Generationcited by 15
  12. 2025"What's Up, Doc?": Analyzing How Users Seek Health Information in Large-Scale Conversational AI Datasetscited by 5
  13. 2026One prompt is not enough: Instruction Sensitivity Undermines Embedding Model Evaluationcited by 3
  14. 2025The Impossibility of Inverse Permutation Learning in Transformer Modelscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Burn After Reading: Do Multimodal Large Language Models Truly Capture Order of Events in Image Sequences?
  2. 2026Evaluating Stochastic Collapse and Implicit Bias in Multimodal Large Language Models
  3. 2026Instinct vs. Reflection: Unifying Token and Verbalized Confidence in Multimodal Large Models
  4. 2025Unexplored flaws in multiple-choice VQA make benchmarking unreliable
  5. 2025Reasoning Multimodal Large Language Model: Data Contamination and Dynamic Evaluation
  6. 2025Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approach
  7. 2026Robust Checkpoint Selection for Multimodal LLMs via Agentic Evaluation and Stability-Aware Ranking
  8. 2024Order Matters in Hallucination: Reasoning Order as Benchmark and Reflexive Prompting for Large-Language-Models