mlsanityData to 7 Oct 2026

Data-Efficient Language-Supervised Zero-Shot Learning with Self-Distillation

arXiv 2104.08945 ↗First posted 18 Apr 2021cs.CVcited by 22

Not on the map. Why?

Abstract (arXiv)

Traditional computer vision models are trained to predict a fixed set of predefined categories. Recently, natural language has been shown to be a broader and richer source of supervision that provides finer descriptions to visual concepts than supervised "gold" labels. Previous works, such as CLIP, use a simple pretraining task of predicting the pairings between images and text captions. CLIP, however, is data hungry and requires more than 400M image text pairs for training. We propose a data-efficient contrastive distillation method that uses soft labels to learn from noisy image-text pairs. Our model transfers knowledge from pretrained image and sentence encoders and achieves strong performance with only 3M image text pairs, 133x smaller than CLIP. Our method exceeds the previous SoTA of general zero-shot learning on ImageNet 21k+1k by 73% relatively with a ResNet50 image encoder and DeCLUTR text encoder. We also beat CLIP by 10.5% relatively on zero-shot evaluation on Google Open Images (19,958 classes).

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 32 of its 44 references that are in the map. First posted on arXiv 18 Apr 2021.

Builds on

32 of 44 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  10. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  11. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  12. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
20 more
  1. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  5. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  6. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  7. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  8. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  9. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  10. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  11. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  12. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
  13. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  14. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  15. 2014Evaluation of Output Embeddings for Fine-Grained Image Classificationcited by 166
  16. 2018Zero-shot Recognition via Semantic Embeddings and Knowledge Graphscited by 155
  17. 2013Zero-Shot Learning by Convex Combination of Semantic Embeddingscited by 135
  18. 2020Learning Visual Representations with Caption Annotationscited by 123
  19. 2018Rethinking Knowledge Graph Propagation for Zero-Shot Learningcited by 87
  20. 2018Label Refinery: Improving ImageNet Classification through Label Progressioncited by 65

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 8; 2023: 8; 2024: 0; 2025: 2; 2026: 1.

22 papers.

Sort:

By area:

  1. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  2. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  3. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  4. 2022MILAN: Masked Image Pretraining on Language Assisted Representationcited by 69
  5. 2022CLIP-TSA: CLIP-Assisted Temporal Self-Attention for Weakly-Supervised Video Anomaly Detectioncited by 67
  6. 2023Composed Image Retrieval using Contrastive Learning and Task-oriented CLIP-based Featurescited by 52
  7. 2022ProposalCLIP: Unsupervised Open-Category Object Proposal Generation via Exploiting CLIP Cuescited by 42
  8. 2023Towards Calibrated Robust Fine-Tuning of Vision-Language Modelscited by 35
  9. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Taskscited by 32
  10. 2021A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervisioncited by 19
  11. 2023Not Only Generative Art: Stable Diffusion for Content-Style Disentanglement in Art Analysiscited by 18
  12. 2023KD-DLGAN: Data Limited Image Generation via Knowledge Distillationcited by 16
10 more
  1. 2023Dual Intent Enhanced Graph Neural Network for Session-based New Item Recommendationcited by 14
  2. 2022Unpaired Referring Expression Grounding via Bidirectional Cross-Modal Matchingcited by 7
  3. 2023Efficient Backdoor Attacks for Deep Neural Networks in Real-world Scenarioscited by 7
  4. 2022Learning by Hallucinating: Vision-Language Pre-training with Weak Supervisioncited by 6
  5. 2025Targeted Forgetting of Image Subgroups in CLIP Modelscited by 6
  6. 2023MXM-CLR: A Unified Framework for Contrastive Learning of Multifold Cross-Modal Representationscited by 3
  7. 2025EcoWikiRS: Learning Ecological Representation of Satellite Images from Weak Supervision with Species Observations and Wikipediacited by 3
  8. 2023Improving Diversity in Zero-Shot GAN Adaptation with Semantic Variationscited by 1
  9. 2026Mean Masked Autoencoder with Flow-Mixing for Encrypted Traffic Classificationcited by 1
  10. 2022Design of the topology for contrastive visual-textual alignmentcited by 0