mlsanityData to 14 Oct 2026

Diagnosing and Rectifying Vision Models using Language

arXiv 2302.04269 ↗First posted 8 Feb 2023cs.LG cs.AI cs.CLcited by 58

Not on the map. Why?

Abstract (arXiv)

Recent multi-modal contrastive learning models have demonstrated the ability to learn an embedding space suitable for building strong vision classifiers, by leveraging the rich information in large-scale image-caption datasets. Our work highlights a distinct advantage of this multi-modal embedding space: the ability to diagnose vision classifiers through natural language. The traditional process of diagnosing model behaviors in deployment settings involves labor-intensive data acquisition and annotation. Our proposed method can discover high-error data slices, identify influential attributes and further rectify undesirable model behaviors, without requiring any visual data. Through a combination of theoretical explanation and empirical verification, we present conditions under which classifiers trained on embeddings from one modality can be equivalently applied to embeddings from another modality. On a range of image datasets with known error slices, we demonstrate that our method can effectively identify the error slices and influential attributes, and can further use language to rectify failure modes of the classifier.

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Builds on 36 of its 49 references that are in the map. First posted on arXiv 8 Feb 2023.

Builds on

36 of 49 references.

References · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  5. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  6. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  8. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  9. 2017Learning Important Features Through Propagating Activation Differencescited by 1,458
  10. 2018Datasheets for Datasetscited by 1,386
  11. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  12. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
24 more
  1. 2018Model Cards for Model Reportingcited by 1,080
  2. 2020Concept Bottleneck Modelscited by 1,032
  3. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  4. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  5. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 629
  6. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  7. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
  8. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  9. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 523
  10. 2017Interpretation of Neural Networks is Fragilecited by 444
  11. 2020Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditingcited by 384
  12. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
  13. 2020No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problemscited by 209
  14. 2021A Fine-Grained Analysis on Distribution Shiftcited by 171
  15. 2022Domino: Discovering Systematic Errors with Cross-Modal Embeddingscited by 156
  16. 2020Fair Attribute Classification through Latent Space De-biasingcited by 125
  17. 2022"This is my unicorn, Fluffy": Personalizing frozen vision-language representationscited by 95
  18. 2022Distilling Model Failures as Directions in Latent Spacecited by 82
  19. 2018Automated Data Slicing for Model Validation:A Big data - AI Integration Approachcited by 70
  20. 2021The Spotlight: A General Method for Discovering Systematic Errors in Deep Learning Modelscited by 63
  21. 2020Understanding Failures of Deep Networks via Robust Feature Extractioncited by 56
  22. 2021A Review on Explainability in Multimodal Deep Neural Netscited by 46
  23. 2022Missingness Bias in Model Debuggingcited by 38
  24. 20213DB: A Framework for Debugging Computer Vision Modelscited by 31

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 21; 2025: 19; 2026: 8.

58 papers.

Sort:

By area:

  1. 2023PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalizationcited by 59
  2. 2024Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Modelscited by 57
  3. 2023Mitigating Spurious Correlations in Multi-modal Models during Fine-tuningcited by 47
  4. 2023LANCE: Stress-testing Visual Models by Generating Language-guided Counterfactual Imagescited by 46
  5. 2025Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversioncited by 43
  6. 2023Mass-Producing Failures of Multimodal Systems with Language Modelscited by 36
  7. 2024Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotationscited by 30
  8. 2024Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Datacited by 27
  9. 2023FACTS: First Amplify Correlations and Then Slice to Discover Biascited by 26
  10. 2024Explaining and Mitigating the Modality Gap in Contrastive Multimodal Learningcited by 26
  11. 2025Post-pre-training for Modality Alignment in Vision-Language Foundation Modelscited by 18
  12. 2024Improving Medical Multi-modal Contrastive Learning with Expert Annotationscited by 16
38 more
  1. 2023LOVM: Language-Only Vision Model Selectioncited by 12
  2. 2023On the Powerfulness of Textual Outlier Exposure for Visual OoD Detectioncited by 12
  3. 2025Closing the Modality Gap for Mixed Modality Searchcited by 12
  4. 2026Cross-Modal Redundancy and the Geometry of Vision-Language Embeddingscited by 10
  5. 2023TextManiA: Enriching Visual Feature by Text-driven Manifold Augmentationcited by 9
  6. 2025Project-Probe-Aggregate: Efficient Fine-Tuning for Group Robustnesscited by 9
  7. 2024Clarify: Improving Model Robustness With Natural Language Correctionscited by 8
  8. 2024Aligning Sight and Sound: Advanced Sound Source Localization Through Audio-Visual Alignmentcited by 8
  9. 2024LADDER: Language-Driven Slice Discovery and Error Rectification in Vision Classifierscited by 7
  10. 2025SpurLens: Automatic Detection of Spurious Cues in Multimodal LLMscited by 7
  11. 2024Concept-based Analysis of Neural Networks via Vision-Language Modelscited by 6
  12. 2024Robo-MUTUAL: Robotic Multimodal Task Specification via Unimodal Learningcited by 6
  13. 2024Sound2Vision: Generating Diverse Visuals from Audio through Cross-Modal Latent Alignmentcited by 5
  14. 2026Closing the Modality Gap Aligns Group-Wise Semanticscited by 5
  15. 2026Is the Modality Gap a Bug or a Feature? A Robustness Perspectivecited by 5
  16. 2024Say My Name: a Model's Bias Discovery Frameworkcited by 4
  17. 2024Unsupervised Model Diagnosiscited by 4
  18. 2025BatStyler: Advancing Multi-category Style Generation for Source-free Domain Generalizationcited by 4
  19. 2024Language Model as Visual Explainercited by 3
  20. 2025Diffusing DeBias: Synthetic Bias Amplification for Model Debiasingcited by 3
  21. 2025Debugging and Runtime Analysis of Neural Networks with VLMs (A Case Study)cited by 3
  22. 2025Q-CLIP: Unleashing the Power of Vision-Language Models for Video Quality Assessment through Unified Cross-Modal Adaptationcited by 3
  23. 2023TLDR: Text Based Last-layer Retraining for Debiasing Image Classifierscited by 2
  24. 2024Embracing Diversity: Interpretable Zero-shot classification beyond one vector per classcited by 2
  25. 2025Superclass-Guided Representation Disentanglement for Spurious Correlation Mitigationcited by 2
  26. 2025Understanding Hardness of Vision-Language Compositionality from A Token-level Causal Lenscited by 2
  27. 2025A Retrospect to Multi-prompt Learning across Vision and Languagecited by 2
  28. 2026TextME: Bridging Unseen Modalities Through Text Descriptionscited by 2
  29. 2023Language-assisted Vision Model Debugger: A Sample-Free Approach to Finding and Fixing Bugscited by 1
  30. 2024Common-Sense Bias Modeling for Classification Taskscited by 1
  31. 2024Cross-modality debiasing: using language to mitigate sub-population shifts in imagingcited by 1
  32. 2024Model Debiasing by Learnable Data Augmentationcited by 1
  33. 2024Federated Vision-Language-Recommendation with Personalized Fusioncited by 1
  34. 2024Attention Head Purification: A New Perspective to Harness CLIP for Domain Generalizationcited by 1
  35. 2025CUBIC: Concept Embeddings for Unsupervised Bias Identification using VLMscited by 1
  36. 2025Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language Modelscited by 1
  37. 2024Refining Skewed Perceptions in Vision-Language Contrastive Models through Visual Representationscited by 0
  38. 2025Learning from Silence and Noise for Visual Sound Source Localizationcited by 0

50 most cited of 58 All 58 (JSON)