mlsanityData to 14 Oct 2026

PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Models

arXiv 2506.14808 ↗First posted 3 Jun 2025cs.LGcited by 3

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision language models (VLMs) respond to user-crafted text prompts and visual inputs, and are applied to numerous real-world problems. VLMs integrate visual modalities with large language models (LLMs), which are well known to be prompt-sensitive. Hence, it is crucial to determine whether VLMs inherit this instability to varying prompts. We therefore investigate which prompt variations VLMs are most sensitive to and which VLMs are most agnostic to prompt variations. To this end, we introduce PARC (Prompt Analysis via Reliability and Calibration), a VLM prompt sensitivity analysis framework built on three pillars: (1) plausible prompt variations in both the language and vision domain, (2) a novel model reliability score with built-in guarantees, and (3) a calibration step that enables dataset- and prompt-spanning prompt variation analysis. Regarding prompt variations, PARC's evaluation shows that VLMs mirror LLM language prompt sensitivity in the vision domain, and most destructive variations change the expected answer. Regarding models, outstandingly robust VLMs among 22 evaluated models come from the InternVL2 family. We further find indications that prompt sensitivity is linked to training data. The code will be at https://github.com/NVlabs/PARC.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 75 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

53 of 75 references.

References · 53

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  7. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  10. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  11. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  12. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
38 more
  1. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  2. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  3. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  4. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  5. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 792
  6. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  7. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  8. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  9. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 454
  10. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  11. 2019Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Comingcited by 336
  12. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
  13. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 308
  14. 2016Least Ambiguous Set-Valued Classifiers with Bounded Error Levelscited by 272
  15. 2024Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillationcited by 249
  16. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
  17. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 216
  18. 2023Conformal Prediction with Large Language Models for Multi-Choice Question Answeringcited by 139
  19. 2024Benchmarking LLMs via Uncertainty Quantificationcited by 126
  20. 20223D Common Corruptions and Data Augmentationcited by 119
  21. 2021Learning to Predict Visual Attributes in the Wildcited by 96
  22. 2024Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvementscited by 83
  23. 2021Understanding by Understanding Not: Modeling Negation in Language Modelscited by 73
  24. 2022Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Studycited by 66
  25. 2023Evaluating the Zero-shot Robustness of Instruction-tuned Language Modelscited by 66
  26. 2020Fashionpedia: Ontology, Segmentation, and an Attribute Localization Datasetcited by 57
  27. 2024IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representationscited by 38
  28. 2022Robustness of Learning from Task Instructionscited by 35
  29. 2024How Susceptible are LLMs to Influence in Prompts?cited by 33
  30. 2023Exploring the Sensitivity of LLMs' Decision-Making Capabilities: Insights from Prompt Variation and Hyperparameterscited by 32
  31. 2024Consistency and Uncertainty: Identifying Unreliable Responses From Black-Box Vision-Language Models for Selective Visual Question Answeringcited by 32
  32. 2024MLLM-CompBench: A Comparative Reasoning Benchmark for Multimodal LLMscited by 32
  33. 2023Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Modelscited by 31
  34. 2024MultiMedEval: A Benchmark and a Toolkit for Evaluating Medical Vision-Language Modelscited by 29
  35. 2024Uncertainty-Aware Evaluation for Vision-Language Modelscited by 29
  36. 2023Sensitivity and Robustness of Large Language Models to Prompt Template in Japanese Text Classification Taskscited by 25
  37. 2022Robustness Analysis of Video-Language Models Against Visual and Language Perturbationscited by 23
  38. 2023PromptAid: Prompt Exploration, Perturbation, Testing and Iteration using Visual Analytics for Large Language Modelscited by 19

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Geometric Risk Control for Vision-Language Model OCRcited by 1
  2. 2026Prompt-Induced Score Variance in Zero-Shot Binary Vision-Language Safety Classificationcited by 1
  3. 2026Target-Checked Reliability Score Refinement for Video Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Modeling Variants of Prompts for Vision-Language Models
  2. 2024Sensitivity of Generative VLMs to Semantically and Lexically Altered Prompts
  3. 2025VP-Bench: A Comprehensive Benchmark for Visual Prompting in Multimodal Large Language Models
  4. 2023Language Models as Black-Box Optimizers for Vision-Language Models
  5. 2024VHELM: A Holistic Evaluation of Vision Language Models
  6. 2024Sycophancy in Vision-Language Models: A Systematic Analysis and an Inference-Time Mitigation Framework
  7. 2025Test-Time Consistency in Vision Language Models
  8. 2023Towards Robust Prompts on Vision-Language Models