mlsanityData to 14 Oct 2026

Promptception: How Sensitive Are Large Multimodal Models to Prompts?

arXiv 2509.03986 ↗First posted 4 Sep 2025cs.CV cs.AI cs.CLcited by 13

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Despite the success of Large Multimodal Models (LMMs) in recent years, prompt design for LMMs in Multiple-Choice Question Answering (MCQA) remains poorly understood. We show that even minor variations in prompt phrasing and structure can lead to accuracy deviations of up to 15% for certain prompts and models. This variability poses a challenge for transparent and fair LMM evaluation, as models often report their best-case performance using carefully selected prompts. To address this, we introduce Promptception, a systematic framework for evaluating prompt sensitivity in LMMs. It consists of 61 prompt types, spanning 15 categories and 6 supercategories, each targeting specific aspects of prompt formulation, and is used to evaluate 10 LMMs ranging from lightweight open-source models to GPT-4o and Gemini 1.5 Pro, across 3 MCQA benchmarks: MMStar, MMMU-Pro, MVBench. Our findings reveal that proprietary models exhibit greater sensitivity to prompt phrasing, reflecting tighter alignment with instruction semantics, while open-source models are steadier but struggle with nuanced and complex phrasing. Based on this analysis, we propose Prompting Principles tailored to proprietary and open-source LMMs, enabling more robust and fair model evaluation.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 38 references that are in the map. First posted on arXiv 4 Sep 2025.

Builds on

32 of 38 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  3. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  4. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phone3 Aug
  5. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2015VQA: Visual Question Answeringcited by 2,506
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  4. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  5. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  6. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  7. 2016A Diagram Is Worth A Dozen Imagescited by 804
  8. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  9. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  10. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 406
  11. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 203
  12. 2022The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrinkcited by 184
14 more
  1. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 182
  2. 2024ProSA: Assessing and Understanding the Prompt Sensitivity of LLMscited by 149
  3. 2024Does Prompt Formatting Have Any Impact on LLM Performance?cited by 141
  4. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 137
  5. 2024What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineeringcited by 90
  6. 2024Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?cited by 71
  7. 2024VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videoscited by 51
  8. 2025Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluationcited by 30
  9. 2024Meta-Prompting for Automating Zero-shot Visual Recognition with LLMscited by 29
  10. 2023Why Is Prompt Tuning for Vision-Language Models Robust to Noisy Labels?cited by 21
  11. 2025DOVE: A Large-Scale Multi-Dimensional Predictions Dataset Towards Meaningful LLM Evaluationcited by 21
  12. 2023Investigating Prompting Techniques for Zero- and Few-Shot Visual Question Answeringcited by 14
  13. 2025Prompt Sentiment: The Catalyst for LLM Changecited by 10
  14. 2024Sensitivity of Generative VLMs to Semantically and Lexically Altered Promptscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 12.

13 papers.

Sort:

By area:

  1. 2026Thought Virus: Viral Misalignment via Subliminal Prompting in Multi-Agent Systemscited by 9
  2. 2025Taming Object Hallucinations with Verified Atomic Confidence Estimationcited by 4
  3. 2026Same Patient, Different Words, Different Diagnosis? Evaluating Semantic Stability in Clinical LLMscited by 2
  4. 2026Prompt-Induced Score Variance in Zero-Shot Binary Vision-Language Safety Classificationcited by 1
  5. 2026CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Modelscited by 1
  6. 2026Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Modelscited by 1
  7. 2026Prompt Robustness Is Task-Dependent: Comparing Objective and Belief-Style Questions in LLM Evaluationcited by 1
  8. 2026MIRROR: Learning from the Other View for Multi-Modal Reasoningcited by 1
  9. 2026PEEM: Prompt Engineering Evaluation Metrics for Interpretable Joint Evaluation of Prompts and Responsescited by 0
  10. 2026Prompts Without Evidence: How Neuroimaging Mentions Shift Clinical Vision-Language Model Predictionscited by 0
  11. 2026Towards Annotation-Free Validation of MLLMs: A Vision-Language Logical Consistency Metriccited by 0
  12. 2026Ask Twice, Look Twice: Prompt Echoing Resolves the Question-First Paradox in Vision-Language Modelscited by 0
1 more
  1. 2026How Do Prompt Variations Affect Energy Consumption in On-Device LLMs?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ReasonLab: A Controlled and Auditable Evaluation of Prompting Techniques for Multiple-Choice QA
  2. 2025Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs
  3. 2025Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs
  4. 2025The Future of MLLM Prompting is Adaptive: A Comprehensive Experimental Evaluation of Prompt Engineering Methods for Robust Multimodal Performance
  5. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications
  6. 2024How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Prompts
  7. 2026When Chain-of-Thought Backfires: Evaluating Prompt Sensitivity in Medical Language Models
  8. 2026Small Language Model Helps Resolve Semantic Ambiguity of LLM Prompt