mlsanityData to 14 Oct 2026

Learning Invariant Causal Mechanism from Vision-Language Models

arXiv 2405.15289 ↗First posted 24 May 2024cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Contrastive Language-Image Pretraining (CLIP) has achieved remarkable success, but its performance can degrade when fine-tuned in out-of-distribution (OOD) scenarios. We model the prediction process using a Structural Causal Model (SCM) and show that the causal mechanism involving both invariant and variant factors in training environments differs from that in test environments. In contrast, the causal mechanism with solely invariant factors remains consistent across environments. We theoretically prove the existence of a linear mapping from CLIP embeddings to invariant factors, which can be estimated using interventional data. Additionally, we provide a condition to guarantee low OOD risk of the invariant predictor. Based on these insights, we propose the Invariant Causal Mechanism of CLIP (CLIP-ICM) framework. CLIP-ICM involves collecting interventional data, estimating a linear projection matrix, and making predictions within the invariant subspace. Experiments on several OOD datasets show that CLIP-ICM significantly improves the performance of CLIP. Our method offers a simple but powerful enhancement, boosting the reliability of CLIP in real-world applications.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 57 of its 85 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

57 of 85 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  2. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  3. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  7. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  8. 2019Invariant Risk Minimizationcited by 1,854
  9. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  10. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  11. 2019Natural Adversarial Examplescited by 1,408
  12. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
38 more
  1. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  2. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,188
  3. 2020In Search of Lost Domain Generalizationcited by 1,054
  4. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  5. 2017Deeper, Broader and Artier Domain Generalizationcited by 986
  6. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 941
  7. 2018Recognition in Terra Incognitacited by 658
  8. 2015Causal inference using invariant prediction: identification and confidence intervalscited by 536
  9. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 454
  10. 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 327
  11. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 312
  12. 2016Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICAcited by 273
  13. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 249
  14. 2020Weakly-Supervised Disentanglement Without Compromisescited by 239
  15. 2013Zero-Shot Learning Through Cross-Modal Transfercited by 237
  16. 2021Contrastive Learning Inverts the Data Generating Processcited by 221
  17. 2021Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalizationcited by 220
  18. 2020Invariant Risk Minimization Gamescited by 188
  19. 2020Improve Unsupervised Domain Adaptation with Mixup Trainingcited by 185
  20. 2020Adaptive Risk Minimization: Learning to Adapt to Domain Shiftcited by 183
  21. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  22. 2021Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICAcited by 160
  23. 2022Interventional Causal Representation Learningcited by 142
  24. 2013Zero-Shot Learning by Convex Combination of Semantic Embeddingscited by 135
  25. 2021On Calibration and Out-of-domain Generalizationcited by 134
  26. 2022Domain Generalization by Mutual-Information Regularization with Pre-trained Modelscited by 133
  27. 2018Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional Robustnesscited by 108
  28. 2020On Linear Identifiability of Learned Representationscited by 105
  29. 2021Model-Based Domain Generalizationcited by 104
  30. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitationscited by 100
  31. 2023CLIPood: Generalizing CLIP to Out-of-Distributionscited by 96
  32. 2023Learning Linear Causal Representations from Interventions under General Nonlinear Mixingcited by 92
  33. 2022Linear Causal Disentanglement via Interventionscited by 84
  34. 2015External Validity: From Do-Calculus to Transportability Across Populationscited by 75
  35. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
  36. 2023Identifiability Results for Multimodal Contrastive Learningcited by 61
  37. 2020Estimating Generalization under Distribution Shifts via Domain-Invariant Representationscited by 56
  38. 2021Domain Prompt Learning for Efficiently Adapting CLIP to Unseen Domainscited by 39

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Enhancing Reward Models for High-quality Image Generation: Beyond Text-Image Alignmentcited by 14
  2. 2026Environment-Invariant Subspace Learning for Generalizable Deepfake Detectioncited by 0
  3. 2026Partition-Aware Unlearning for Removing Spurious Correlations in Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning
  2. 2025Post-pre-training for Modality Alignment in Vision-Language Foundation Models
  3. 2023A Closer Look at the Explainability of Contrastive Language-Image Pre-training
  4. 2024Diffusion Feedback Helps CLIP See Better
  5. 2024Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations
  6. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks
  7. 2023Improved baselines for vision-language pre-training
  8. 2024Refining Skewed Perceptions in Vision-Language Contrastive Models through Visual Representations