mlsanityData to 7 Oct 2026

Enhancing CLIP Robustness via Cross-Modality Alignment

arXiv 2510.24038 ↗First posted 28 Oct 2025cs.CV cs.MAcited by 14

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-language models (VLMs) such as CLIP demonstrate strong generalization in zero-shot classification but remain highly vulnerable to adversarial perturbations. Existing methods primarily focus on adversarial fine-tuning or prompt optimization; they often overlook the gaps in CLIP's encoded features, which is shown as the text and image features lie far apart from each other. This misalignment is significantly amplified under adversarial perturbations, leading to severe degradation in classification performance. To address this problem, we propose Cross-modality Alignment, dubbed COLA, an optimal transport-based framework that explicitly addresses adversarial misalignment by restoring both global image-text alignment and local structural consistency in the feature space. (1) COLA first projects adversarial image embeddings onto a subspace spanned by class text features, effectively filtering out non-semantic distortions while preserving discriminative information. (2) It then models images and texts as discrete distributions over multiple augmented views and refines their alignment via OT, with the subspace projection seamlessly integrated into the cost computation. This design ensures stable cross-modal alignment even under adversarial conditions. COLA is training-free and compatible with existing fine-tuned models. Extensive evaluations across 14 zero-shot classification benchmarks demonstrate the effectiveness of COLA, especially with an average improvement of 6.7% on ImageNet and its variants under PGD adversarial attacks, while maintaining high accuracy on clean samples.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 69 references that are in the map. First posted on arXiv 28 Oct 2025.

Builds on

53 of 69 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Dynamic Multimodal Prototype Learning in Vision-Language Models4 Jul
  2. 2024Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting25 Oct
  3. 2024Selective Vision-Language Subspace Projection for Few-shot CLIP24 Jul
  4. 2024Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models5 Jun
  5. 2023Tuning Multi-mode Token-level Prompt Alignment across Modalities25 Sep
  6. 2023Waffling around for Performance: Visual Classification with Random Words and Broad Concepts12 Jun

Other references · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  3. 2013Intriguing properties of neural networkscited by 5,114
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  6. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  7. 2013Describing Textures in the Wildcited by 2,110
  8. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  9. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  10. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  11. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  12. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
35 more
  1. 2019Natural Adversarial Examplescited by 1,401
  2. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  3. 2020Fast is better than free: Revisiting adversarial trainingcited by 723
  4. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  5. 2017Countering Adversarial Images using Input Transformationscited by 517
  6. 2020Overfitting in adversarially robust deep learningcited by 504
  7. 2022Diffusion Models for Adversarial Purificationcited by 469
  8. 2015Optimal Transport for Domain Adaptationcited by 457
  9. 2021Recent Advances in Adversarial Training for Adversarial Robustnesscited by 276
  10. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  11. 2024AlphaEdit: Null-Space Constrained Knowledge Editing for Language Modelscited by 224
  12. 2021Adversarial purification with Score-based generative modelscited by 132
  13. 2018Graph-Based Global Reasoning Networkscited by 120
  14. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Modelscited by 119
  15. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Modelscited by 117
  16. 2019Scalable Gromov-Wasserstein Learning for Graph Partitioning and Matchingcited by 102
  17. 2022Evaluating the Adversarial Robustness of Adaptive Test-time Defensescited by 64
  18. 2024Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustnesscited by 48
  19. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Modelscited by 48
  20. 2021Adversarial Attacks are Reversible with Natural Supervisioncited by 47
  21. 2023Adversarial Prompt Tuning for Vision-Language Modelscited by 42
  22. 2020Iterative label cleaning for transductive and semi-supervised few-shot learningcited by 40
  23. 2023Generalized Logit Adjustment: Calibrating Fine-tuned Models by Removing Label Bias in Foundation Modelscited by 32
  24. 2021Attacking Adversarial Attacks as A Defensecited by 31
  25. 2025SafeMLRM: Demystifying Safety in Multi-modal Large Reasoning Modelscited by 29
  26. 2021Combating Adversaries with Anti-Adversariescited by 27
  27. 2021Enhancing Adversarial Robustness via Test-time Transformation Ensemblingcited by 27
  28. 2023Language-Driven Anchors for Zero-Shot Adversarial Robustnesscited by 23
  29. 2025CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIPcited by 23
  30. 2025Accelerating Diffusion Transformer via Gradient-Optimized Cachecited by 15
  31. 2022Adaptive Distribution Calibration for Few-Shot Learning with Hierarchical Optimal Transportcited by 14
  32. 2025Accelerating Diffusion Transformer via Error-Optimized Cachecited by 14
  33. 2024Boosting Few-Shot Learning via Attentive Feature Regularizationcited by 10
  34. 2025Project-Probe-Aggregate: Efficient Fine-Tuning for Group Robustnesscited by 9
  35. 2024Robust Fine-tuning of Zero-shot Models via Variance Reductioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 14.

14 papers.

Sort:

By area:

  1. 2026Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigationcited by 7
  2. 2026Revisiting Robustness for LLM Safety Alignment via Selective Geometry Controlcited by 3
  3. 2026Mitigating Hallucinations in Large Vision-Language Models without Performance Degradationcited by 3
  4. 2026Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIPcited by 2
  5. 2026GuardAlign: Test-time Safety Alignment in Multimodal Large Language Modelscited by 1
  6. 2026Reevaluating the Intra-Modal Misalignment Hypothesis in CLIPcited by 1
  7. 2026When CLIP Sees More, It Fights Back Harder: Multi-View Guided Adaptive Counterattacks for Test-Time Adversarial Robustnesscited by 1
  8. 2026Beyond Heuristic Prompting: A Concept-Guided Bayesian Framework for Zero-Shot Image Recognitioncited by 0
  9. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Modelscited by 0
  10. 2026Adapting Point Cloud Analysis via Multimodal Bayesian Distribution Learningcited by 0
  11. 2026Your CLIP has 164 dimensions of noise: Exploring the embeddings covariance eigenspectrum of contrastively pretrained vision-language transformerscited by 0
  12. 2026T-VSS: Test-Time Visual Subspace Steering for Adversarial Robustness of Vision-Language Modelscited by 0
2 more
  1. 2026From Inference to Adaptation: A Unified Optimal Transport View of Vision Language Modelcited by 0
  2. 2026PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Bridging the Task Gap: Multi-Task Adversarial Transferability in CLIP and Its Derivatives
  2. 2025Modest-Align: Data-Efficient Alignment for Vision-Language Models
  3. 2026AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Models
  4. 2025Post-pre-training for Modality Alignment in Vision-Language Foundation Models
  5. 2025Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models
  6. 2026Investigating Adversarial Robustness of Multi-modal Large Language Models
  7. 2026Cross-Modal Prototype Alignment and Mixing for Training-Free Few-Shot Classification
  8. 2024ComAlign: Compositional Alignment in Vision-Language Models