OTTER: Effortless Label Distribution Adaptation of Zero-shot Models
Not on the map. Why?
Abstract (arXiv)
Popular zero-shot models suffer due to artifacts inherited from pretraining. One particularly detrimental issue, caused by unbalanced web-scale pretraining data, is mismatched label distribution. Existing approaches that seek to repair the label distribution are not suitable in zero-shot settings, as they have mismatching requirements, such as needing access to labeled downstream task data or knowledge of the true label balance in the pretraining distribution. We sidestep these challenges and introduce a simple and lightweight approach to adjust pretrained model predictions via optimal transport. Our technique requires only an estimate of the label distribution of a downstream task. Theoretically, we characterize the improvement produced by our procedure under certain mild conditions and provide bounds on the error caused by misspecification. Empirically, we validate our method in a wide array of zero-shot image and text classification tasks, improving accuracy by 4.8% and 15.9% on average, and beating baselines like prior matching -- often by significant margins -- in 17 out of 21 datasets.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 40 of its 72 references that are in the map. First posted on arXiv 12 Apr 2024.
Builds on
40 of 72 references.
References · 40
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2023Visual Instruction Tuningcited by 7,792
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
- 2021Learning to Prompt for Vision-Language Modelscited by 2,269
- 2013Describing Textures in the Wildcited by 2,110
- 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
- 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
28 more
- 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
- 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
- 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
- 2020HateXplain: A Benchmark Dataset for Explainable Hate Speech Detectioncited by 403
- 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 396
- 2018Detecting and Correcting for Label Shift with Black Box Predictorscited by 364
- 2022Visual Classification via Description from Large Language Modelscited by 331
- 2023Debiasing Vision-Language Models via Biased Promptscited by 147
- 2021Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implicationscited by 128
- 2020Domain Adaptation with Conditional Distribution Matching and Generalized Label Shiftcited by 124
- 2020A Unified View of Label Shift Estimationcited by 119
- 2019Regularized Learning for Domain Adaptation under Label Shiftscited by 118
- 2020Multi-Dimensional Gender Bias Classificationcited by 80
- 2023CHiLS: Zero-Shot Image Classification with Hierarchical Label Setscited by 79
- 2022Unified Optimal Transport Framework for Universal Domain Adaptationcited by 57
- 2022Discovering Latent Concepts Learned in BERTcited by 56
- 2022Fair and Optimal Classification via Post-Processingcited by 50
- 2021Sinkhorn Label Allocation: Semi-Supervised Classification via Annealed Self-Trainingcited by 32
- 2023Generalized Logit Adjustment: Calibrating Fine-tuned Models by Removing Label Bias in Foundation Modelscited by 32
- 2022SoLar: Sinkhorn Label Refinery for Imbalanced Partial-Label Learningcited by 30
- 2023Zero-Shot Robustification of Zero-Shot Modelscited by 24
- 2022Learning to Re-weight Examples with Optimal Transport for Imbalanced Classificationcited by 23
- 2022Improving Zero-Shot Models with Label Distribution Priorscited by 14
- 2024P2OT: Progressive Partial Optimal Transport for Deep Imbalanced Clusteringcited by 13
- 2019New characterizations of Hoffman constants for systems of linear constraintscited by 9
- 2023Mitigating Word Bias in Zero-shot Prompt-based Classifierscited by 9
- 2023Geometry-Aware Adaptation for Pretrained Modelscited by 5
- 2023An easily computable upper bound on the Hoffman constant for homogeneous inequality systemscited by 3
Built on by
Citing papers per year: 2025: 1.
1 paper.