mlsanityData to 7 Oct 2026

Understanding the Vulnerability of CLIP to Image Compression

arXiv 2311.14029 ↗First posted 23 Nov 2023cs.CV cs.LGcited by 3

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

CLIP is a widely used foundational vision-language model that is used for zero-shot image recognition and other image-text alignment tasks. We demonstrate that CLIP is vulnerable to change in image quality under compression. This surprising result is further analysed using an attribution method-Integrated Gradients. Using this attribution method, we are able to better understand both quantitatively and qualitatively exactly the nature in which the compression affects the zero-shot recognition accuracy of this model. We evaluate this extensively on CIFAR-10 and STL-10. Our work provides the basis to understand this vulnerability of CLIP and can help us develop more effective methods to improve the robustness of CLIP and other vision-language models.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 21 references that are in the map. First posted on arXiv 23 Nov 2023.

Builds on

17 of 21 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Why Is Prompt Tuning for Vision-Language Models Robust to Noisy Labels?22 Jul
  2. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 15

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  4. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  5. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  6. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  7. 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
  8. 2014Striving for Simplicity: The All Convolutional Netcited by 983
  9. 2021Understanding Robustness of Transformers for Image Classificationcited by 250
  10. 2016Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layerscited by 197
  11. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  12. 2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
3 more
  1. 2021Fast Axiomatic Attribution for Neural Networkscited by 35
  2. 2022Robustness Analysis of Video-Language Models Against Visual and Language Perturbationscited by 23
  3. 2022Defending Multimodal Fusion Models against Single-Source Adversariescited by 20

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 1.

3 papers.

  1. 2024TypeScore: A Text Fidelity Metric for Text-to-Image Generative Modelscited by 4
  2. 2024Artificial Intelligence for Collective Intelligence: A National-Scale Research Strategycited by 4
  3. 2025Processing and acquisition traces in visual encoders: What does CLIP know about your camera?cited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2024Toward a Holistic Evaluation of Robustness in CLIP Models
  2. 2025Analyzing CLIP's Performance Limitations in Multi-Object Scenarios: A Controlled High-Resolution Study
  3. 2024The Hard Positive Truth about Vision-Language Compositionality
  4. 2023PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contexts
  5. 2025Uniformity First: Uniformity-aware Test-time Adaptation of Vision-language Models against Image Corruption
  6. 2026Explaining CLIP Zero-shot Predictions Through Concepts
  7. 2023Learning Mask-aware CLIP Representations for Zero-Shot Segmentation
  8. 2024Zoom-shot: Fast and Efficient Unsupervised Zero-Shot Transfer of CLIP to Vision Encoders with Multimodal Loss