# On the Domain Robustness of Contrastive Vision-Language Models

arXiv: https://arxiv.org/abs/2506.23663 · HTML: https://mlsanity.com/paper/2506.23663/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 25 of its 48 references that are in the map.
- First posted on arXiv 30 Jun 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 25 of 25)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [RobustBench: a standardized adversarial robustness benchmark](https://mlsanity.com/paper/2010.09670/) (2020)
- [Measuring Robustness to Natural Distribution Shifts in Image Classification](https://mlsanity.com/paper/2007.00644/) (2020)
- [AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classification](https://mlsanity.com/paper/1608.05167/) (2016)
- [Challenges in Representation Learning: A report on three machine learning contests](https://mlsanity.com/paper/1307.0414/) (2013)
- [A Fourier Perspective on Model Robustness in Computer Vision](https://mlsanity.com/paper/1906.08988/) (2019)
- [Demystifying CLIP Data](https://mlsanity.com/paper/2309.16671/) (2023)
- [Fast AutoAugment](https://mlsanity.com/paper/1905.00397/) (2019)
- [Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)](https://mlsanity.com/paper/2205.01397/) (2022)
- [Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models](https://mlsanity.com/paper/2303.06628/) (2023)
- [BRAVE: Broadening the visual encoding of vision-language models](https://mlsanity.com/paper/2404.07204/) (2024)
- [A Sober Look at the Robustness of CLIPs to Spurious Features](https://mlsanity.com/paper/2403.11497/) (2024)
- [RLSbench: Domain Adaptation Under Relaxed Label Shift](https://mlsanity.com/paper/2302.03020/) (2023)
- [Beyond Human Vision: The Role of Large Vision Language Models in Microscope Image Analysis](https://mlsanity.com/paper/2405.00876/) (2024)

## Built on by (most cited first, 1 of 1)

- [Concept Regions Matter: Benchmarking CLIP with a New Cluster-Importance Approach](https://mlsanity.com/paper/2511.12978/) (2025)

## Abstract (arXiv)

In real-world vision-language applications, practitioners increasingly rely on large, pretrained foundation models rather than custom-built solutions, despite limited transparency regarding their training data and processes. While these models achieve impressive performance on general benchmarks, their effectiveness can decline notably under specialized domain shifts, such as unique imaging conditions or environmental variations. In this work, we introduce Deepbench, a framework designed to assess domain-specific robustness of vision-language models (VLMs). Deepbench leverages a large language model (LLM) to generate realistic, context-aware image corruptions tailored to specific deployment domains without requiring labeled data. We evaluate a range of contrastive vision-language architectures and architectural variants across six real-world domains and observe substantial variability in robustness, highlighting the need for targeted, domain-aware evaluation. Deepbench is released as open-source software to support further research into domain-aware robustness assessment.