mlsanityData to 7 Oct 2026

MM-R3: On (In-)Consistency of Vision-Language Models (VLMs)

arXiv 2410.04778 ↗First posted 7 Oct 2024cs.CVcited by 10

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

With the advent of LLMs and variants, a flurry of research has emerged, analyzing the performance of such models across an array of tasks. While most studies focus on evaluating the capabilities of state-of-the-art (SoTA) Vision Language Models (VLMs) through task accuracy (e.g., visual question answering, grounding), our work explores the related but complementary aspect of consistency - the ability of a VLM to produce semantically similar or identical responses to semantically similar queries. We note that consistency is a fundamental prerequisite (necessary but not sufficient condition) for robustness and trust in VLMs. Armed with this perspective, we propose the MM-R3 benchmark, which allows us to analyze performance, in terms of consistency and accuracy, of SoTA VLMs on three tasks: Question Rephrasing, Image Restyling, and Context Reasoning. Our analysis reveals that consistency does not always align with accuracy, indicating that models with higher accuracy are not necessarily more consistent, and vice versa. Furthermore, we propose a simple yet effective mitigation strategy in the form of an adapter module trained to minimize inconsistency across prompts. With our proposed strategy, we are able to achieve absolute improvements of 5.7% and 12.5%, on average on widely used VLMs such as BLIP-2 and LLaVa 1.5M in terms of consistency over their existing counterparts.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 56 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

37 of 56 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug

Other references · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  7. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2015VQA: Visual Question Answeringcited by 2,489
  11. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
24 more
  1. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  2. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  3. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  5. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  7. 2016Modeling Context in Referring Expressionscited by 1,054
  8. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  9. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
  10. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  11. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  12. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  13. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  14. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 305
  15. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 284
  16. 2021Understanding Robustness of Transformers for Image Classificationcited by 250
  17. 2021TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentationcited by 225
  18. 2023Universal Self-Consistency for Large Language Model Generationcited by 198
  19. 2021Evaluating the Robustness of Neural Language Models to Input Perturbationscited by 87
  20. 2019CLEVR-Ref+: Diagnosing Visual Reasoning with Referring Expressionscited by 73
  21. 2024How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Promptscited by 47
  22. 2022The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoningcited by 38
  23. 2020A Closer Look at the Robustness of Vision-and-Language Pre-trained Modelscited by 37
  24. 2021P-Adapters: Robustly Extracting Factual Information from Language Models with Diverse Promptscited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 7.

10 papers.

  1. 2024Cross-Modal Consistency in Multimodal Large Language Modelscited by 17
  2. 2025Exploring and Evaluating Multimodal Knowledge Reasoning Consistency of Multimodal Large Language Modelscited by 8
  3. 2026Remember with Confidence: Uncertainty Quantification for Spatio-temporal Memory with Probabilistic Guaranteescited by 3
  4. 2026RemoteShield: Enable Robust Multimodal Large Language Models for Earth Observationcited by 2
  5. 2026FUSE: Quantifying Uncertainty in Vision-Language Models by Bayesian Fusing Epistemic and Aleatoric Uncertaintycited by 1
  6. 2026SEER: A Self-Grounded Evidence Interface for Controlled Spatial Relation Classificationcited by 1
  7. 2025Response Wide Shut? Surprising Observations in Basic Vision Language Model Capabilitiescited by 0
  8. 2026Towards Annotation-Free Validation of MLLMs: A Vision-Language Logical Consistency Metriccited by 0
  9. 2026VISTA: Auditing Semantic Divergence in Vision-Language Modelscited by 0
  10. 2026Evaluating the Diagnostic Robustness of Vision-Language Models Under Visual and Textual Perturbationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Test-Time Consistency in Vision Language Models
  2. 2024Unveiling the Tapestry of Consistency in Large Vision-Language Models
  3. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
  4. 2025When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs
  5. 2024CAST: Cross-modal Alignment Similarity Test for Vision Language Models
  6. 2026Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark
  7. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  8. 2026Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models