mlsanityData to 7 Oct 2026

Data Efficient Language-supervised Zero-shot Recognition with Optimal Transport Distillation

arXiv 2112.09445 ↗First posted 17 Dec 2021cs.CVcited by 35

Not on the map. Why?

Abstract (arXiv)

Traditional computer vision models are trained to predict a fixed set of predefined categories. Recently, natural language has been shown to be a broader and richer source of supervision that provides finer descriptions to visual concepts than supervised "gold" labels. Previous works, such as CLIP, use InfoNCE loss to train a model to predict the pairing between images and text captions. CLIP, however, is data hungry and requires more than 400M image-text pairs for training. The inefficiency can be partially attributed to the fact that the image-text pairs are noisy. To address this, we propose OTTER (Optimal TransporT distillation for Efficient zero-shot Recognition), which uses online entropic optimal transport to find a soft image-text match as labels for contrastive learning. Based on pretrained image and text encoders, models trained with OTTER achieve strong performance with only 3M image text pairs. Compared with InfoNCE loss, label smoothing, and knowledge distillation, OTTER consistently outperforms these baselines in zero shot evaluation on Google Open Images (19,958 classes) and multi-labeled ImageNet 10K (10032 classes) from Tencent ML-Images. Over 42 evaluations on 7 different dataset/architecture settings x 6 metrics, OTTER outperforms (32) or ties (2) all baselines in 34 of them.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 64 of its 94 references that are in the map. First posted on arXiv 17 Dec 2021.

Builds on

64 of 94 references.

References · 64

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  9. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  10. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  11. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  12. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
38 more
  1. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  6. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  8. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  9. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  10. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  11. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  12. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  13. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  14. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  15. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  16. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  17. 2019Contrastive Representation Distillationcited by 734
  18. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  19. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  20. 2020Contrastive Learning with Hard Negative Samplescited by 619
  21. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  22. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 580
  23. 2019Self-labelling via simultaneous clustering and representation learningcited by 525
  24. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
  25. 2015Optimal Transport for Domain Adaptationcited by 457
  26. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  27. 2021Unbiased Teacher for Semi-Supervised Object Detectioncited by 305
  28. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  29. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  30. 2019Visual Semantic Reasoning for Image-Text Matchingcited by 185
  31. 2014Evaluation of Output Embeddings for Fine-Grained Image Classificationcited by 166
  32. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  33. 2017Learning from Noisy Labels with Distillationcited by 160
  34. 2018Zero-shot Recognition via Semantic Embeddings and Knowledge Graphscited by 155
  35. 2020FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensionscited by 152
  36. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 136
  37. 2020Learning the Best Pooling Strategy for Visual Semantic Embeddingcited by 136
  38. 2013Zero-Shot Learning by Convex Combination of Semantic Embeddingscited by 135

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 16; 2024: 5; 2025: 5; 2026: 1.

35 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  2. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  3. 2023Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Trainingcited by 91
  4. 2023LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collectionscited by 43
  5. 2022Democratizing Contrastive Language-Image Pre-training: A CLIP Benchmark of Data, Model, and Supervisioncited by 36
  6. 2022On Guiding Visual Attention with Language Specificationcited by 30
  7. 2023S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captionscited by 29
  8. 2022Non-Contrastive Learning Meets Language-Image Pre-Trainingcited by 25
  9. 2022Open-Vocabulary 3D Detection via Image-level Class and Debiased Cross-modal Contrastive Learningcited by 23
  10. 2022Exploring Hierarchical Graph Representation for Large-Scale Zero-Shot Image Classificationcited by 21
  11. 2023Rewrite Caption Semantics: Bridging Semantic Gaps for Language-Supervised Semantic Segmentationcited by 21
  12. 2023Improved baselines for vision-language pre-trainingcited by 20
23 more
  1. 2023Probing Conceptual Understanding of Large Visual-Language Modelscited by 17
  2. 2023Domain Prompt Learning with Quaternion Networkscited by 17
  3. 2023Improving Cross-lingual Information Retrieval on Low-Resource Languages via Optimal Transport Distillationcited by 15
  4. 2023RILS: Masked Visual Reconstruction in Language Semantic Spacecited by 10
  5. 2024Revisiting Deep Audio-Text Retrieval Through the Lens of Transportationcited by 10
  6. 2023TAP: Targeted Prompting for Task Adaptive Generation of Textual Training Instances for Visual Classificationcited by 8
  7. 2023MOFI: Learning Image Representations from Noisy Entity Annotated Imagescited by 7
  8. 2024FFF: Fixing Flawed Foundations in contrastive pre-training results in very strong Vision-Language modelscited by 7
  9. 2024Embedding Geometries of Contrastive Language-Image Pre-Trainingcited by 6
  10. 2025Target Refocusing via Attention Redistribution for Open-Vocabulary Semantic Segmentation: An Explainability Perspectivecited by 5
  11. 2022Universal Prototype Transport for Zero-Shot Action Recognition and Localizationcited by 4
  12. 2023Getting More Juice Out of Your Data: Hard Pair Refinement Enhances Visual-Language Models Without Extra Datacited by 3
  13. 2023Few-Shot Object Detection via Synthetic Features with Optimal Transportcited by 3
  14. 2024Open-Vocabulary Mobile Manipulation Based on Double Relaxed Contrastive Learning with Dense Labelingcited by 3
  15. 2022Correlation between Alignment-Uniformity and Performance of Dense Contrastive Representationscited by 1
  16. 2023Prototypical Contrastive Transfer Learning for Multimodal Language Understandingcited by 1
  17. 2024Semantic Compositions Enhance Vision-Language Contrastive Learningcited by 1
  18. 2025Unlearning the Noisy Correspondence Makes CLIP More Robustcited by 1
  19. 2025Enhancing 3D Medical Image Understanding with Pretraining Aided by 2D Multimodal Large Language Modelscited by 1
  20. 2023ELVIS: Empowering Locality of Vision Language Pre-training with Intra-modal Similaritycited by 0
  21. 2025Weighted Multi-Prompt Learning with Description-free Large Language Model Distillationcited by 0
  22. 2025CLIPin: A Non-contrastive Plug-in to CLIP for Multimodal Semantic Alignmentcited by 0
  23. 2026NaiLIA: Multimodal Nail Design Retrieval Based on Dense Intent Descriptions and Palette Queriescited by 0