mlsanityData to 7 Oct 2026

Cross-modality debiasing: using language to mitigate sub-population shifts in imaging

arXiv 2403.07888 ↗First posted 2 Feb 2024cs.CV cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Sub-population shift is a specific type of domain shift that highlights changes in data distribution within specific sub-groups or populations between training and testing. Sub-population shift accounts for a significant source of algorithmic bias and calls for distributional robustness. Recent studies found inherent distributional robustness in multi-modality foundation models, such as the vision-language model CLIP, yet this robustness is vulnerable through parameter fine-tuning. In this paper, we propose leveraging the connection of robustness among different modalities and reshaping the distributional robustness of one modality with another. Specifically, in the context of the distributional robustness of CLIP, we propose to leverage natural language inputs to debias the image feature representations, to improve worst-case performance on sub-populations. Our extensive empirical studies show that image representations debiased by natural language can achieve significant performance improvement and reduction of performance instability under sub-population shifts.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 31 of its 46 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

31 of 46 references.

References · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  3. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  4. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  5. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  6. 2021Robust fine-tuning of zero-shot modelscited by 785
  7. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  8. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
  9. 2018Fairness Without Demographics in Repeated Loss Minimizationcited by 363
  10. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 334
  11. 2017Certifying Some Distributional Robustness with Principled Adversarial Trainingcited by 333
  12. 2020No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problemscited by 208
19 more
  1. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  2. 2018Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networkscited by 171
  3. 2019Wasserstein Distributionally Robust Optimization: Theory and Applications in Machine Learningcited by 169
  4. 2020BREEDS: Benchmarks for Subpopulation Shiftcited by 155
  5. 2023Debiasing Vision-Language Models via Biased Promptscited by 147
  6. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 137
  7. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  8. 2023Foundation Models and Fair Usecited by 112
  9. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  10. 2018Safe Reinforcement Learning with Model Uncertainty Estimatescited by 70
  11. 2022Contrastive Adapters for Foundation Model Group Robustnesscited by 62
  12. 2023Diagnosing and Rectifying Vision Models using Languagecited by 58
  13. 2021DORO: Distributional and Outlier Robust Optimizationcited by 54
  14. 2022Using Language to Extend to Unseen Domainscited by 34
  15. 2022On Guiding Visual Attention with Language Specificationcited by 30
  16. 2022Understanding Why Generalized Reweighting Does Not Improve Over ERMcited by 29
  17. 2017Addressing Appearance Change in Outdoor Robotics with Adversarial Domain Adaptationcited by 22
  18. 2022FairCLIP: Social Bias Elimination based on Attribute Prototype Learning and Representation Neutralizationcited by 20
  19. 2018A Data-Efficient Framework for Training and Sim-to-Real Transfer of Navigation Policiescited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Density-Aware Translation of Spurious Correlations in Zero-Shot VLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Bias Detection and Rotation-Robustness Mitigation in Vision-Language Models and Generative Image Models
  2. 2025On the Robustness of Medical Vision-Language Models: Are they Truly Generalizable?
  3. 2024A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks
  4. 2025Bias Is a Subspace, Not a Coordinate: A Geometric Rethinking of Post-hoc Debiasing in Vision-Language Models
  5. 2026Scaling Vision-Language Models Is Not Enough to Mitigate Bias
  6. 2026Is the Modality Gap a Bug or a Feature? A Robustness Perspective
  7. 2025On the Domain Robustness of Contrastive Vision-Language Models
  8. 2025Beyond Cross-Modal Alignment: Measuring and Leveraging Modality Gap in Vision-Language Models