mlsanityData to 14 Oct 2026

Rethinking Misalignment in Vision-Language Model Adaptation from a Causal Perspective

arXiv 2410.12816 ↗First posted 1 Oct 2024cs.CV cs.CL cs.LGcited by 9

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Foundational Vision-Language models such as CLIP have exhibited impressive generalization in downstream tasks. However, CLIP suffers from a two-level misalignment issue, i.e., task misalignment and data misalignment, when adapting to specific tasks. Soft prompt tuning has mitigated the task misalignment, yet the data misalignment remains a challenge. To analyze the impacts of the data misalignment, we revisit the pre-training and adaptation processes of CLIP and develop a structural causal model. We discover that while we expect to capture task-relevant information for downstream tasks accurately, the task-irrelevant knowledge impacts the prediction results and hampers the modeling of the true relationships between the images and the predicted classes. As task-irrelevant knowledge is unobservable, we leverage the front-door adjustment and propose Causality-Guided Semantic Decoupling and Classification (CDC) to mitigate the interference of task-irrelevant knowledge. Specifically, we decouple semantics contained in the data of downstream tasks and perform classification based on each semantic. Furthermore, we employ the Dempster-Shafer evidence theory to evaluate the uncertainty of each prediction generated by diverse semantics. Experiments conducted in multiple different settings have consistently demonstrated the effectiveness of CDC.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 32 of its 49 references that are in the map. First posted on arXiv 1 Oct 2024.

Builds on

32 of 49 references.

References · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  3. 2013Describing Textures in the Wildcited by 2,125
  4. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  5. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  6. 2022Visual Prompt Tuningcited by 1,681
  7. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  8. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  9. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  10. 2019Natural Adversarial Examplescited by 1,408
  11. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  12. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
20 more
  1. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  2. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  3. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 362
  4. 2022Prompt-aligned Gradient for Prompt Tuningcited by 341
  5. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  6. 2022Trusted Multi-View Classification with Dynamic Evidential Fusioncited by 168
  7. 2020Interventional Few-Shot Learningcited by 112
  8. 2021Counterfactual Generative Networkscited by 111
  9. 2021Counterfactual Zero-Shot and Open-Set Visual Recognitioncited by 95
  10. 2021Trusted Multi-View Classificationcited by 94
  11. 2023Consistency-guided Prompt Learning for Vision-Language Modelscited by 73
  12. 2023ArGue: Attribute-Guided Prompt Tuning for Vision-Language Modelscited by 39
  13. 2022SgVA-CLIP: Semantic-guided Visual Adapting of Vision-Language Models for Few-shot Image Classificationcited by 34
  14. 2021Towards Robust Classification Model by Counterfactual and Invariant Data Generationcited by 30
  15. 2023Learning Hierarchical Prompt with Structured Linguistic Knowledge for Vision-Language Modelscited by 23
  16. 2021Counterfactual Samples Synthesizing and Training for Robust Visual Question Answeringcited by 22
  17. 2022Causal Balancing for Domain Generalizationcited by 20
  18. 2023APoLLo: Unified Adapter and Prompt Learning for Vision Language Modelscited by 19
  19. 2022Disentangle and Remerge: Interventional Knowledge Distillation for Few-Shot Object Detection from A Conditional Causal Perspectivecited by 13
  20. 2022Interventional Contrastive Learning with Meta Semantic Regularizercited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 2.

9 papers.

  1. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  2. 2025Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Modelscited by 23
  3. 2025Enhancing Reward Models for High-quality Image Generation: Beyond Text-Image Alignmentcited by 14
  4. 2025On the Value of Cross-Modal Misalignment in Multimodal Representation Learningcited by 10
  5. 2025FOCoOp: Enhancing Out-of-Distribution Robustness in Federated Prompt Learning for Vision-Language Modelscited by 2
  6. 2025Causality-guided Prompt Learning for Vision-language Models via Visual Granulationcited by 2
  7. 2025Is Exchangeability better than I.I.D to handle Data Distribution Shifts while Pooling Data for Data-scarce Medical image segmentation?cited by 1
  8. 2026FedMPT: Federated Multi-label Prompt Tuning of Vision-Language Modelscited by 0
  9. 2026Concept Driven Domain Adaptation: Finding an Abstract Needle in a Haystackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CAPT: Confusion-Aware Prompt Tuning for Reducing Vision-Language Misalignment
  2. 2026Beyond Perception Errors: Semantic Fixation in Large Vision-Language Models
  3. 2025V-SEAM: Visual Semantic Editing and Attention Modulating for Causal Interpretability of Vision-Language Models
  4. 2025Test-Time Consistency in Vision Language Models
  5. 2026Dive Into the Implicit Biases of Low-rank Vision-language Alignment
  6. 2024Understanding Retrieval-Augmented Task Adaptation for Vision-Language Models
  7. 2026PERL: Parameter Efficient Reasoning in CLIP Latent Space
  8. 2024Vision-Language Models Create Cross-Modal Task Representations