mlsanityData to 14 Oct 2026

Expediting Contrastive Language-Image Pretraining via Self-distilled Encoders

arXiv 2312.12659 ↗First posted 19 Dec 2023cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Recent advances in vision language pretraining (VLP) have been largely attributed to the large-scale data collected from the web. However, uncurated dataset contains weakly correlated image-text pairs, causing data inefficiency. To address the issue, knowledge distillation have been explored at the expense of extra image and text momentum encoders to generate teaching signals for misaligned image-text pairs. In this paper, our goal is to resolve the misalignment problem with an efficient distillation framework. To this end, we propose ECLIPSE: Expediting Contrastive Language-Image Pretraining with Self-distilled Encoders. ECLIPSE features a distinctive distillation architecture wherein a shared text encoder is utilized between an online image encoder and a momentum image encoder. This strategic design choice enables the distillation to operate within a unified projected space of text embedding, resulting in better performance. Based on the unified text embedding space, ECLIPSE compensates for the additional computational cost of the momentum image encoder by expediting the online image encoder. Through our extensive experiments, we validate that there is a sweet spot between expedition and distillation where the partial view from the expedited online image encoder interacts complementarily with the momentum teacher. As a result, ECLIPSE outperforms its counterparts while achieving substantial acceleration in inference speed.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 48 references that are in the map. First posted on arXiv 19 Dec 2023.

Builds on

40 of 48 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  10. 2020End-to-End Object Detection with Transformerscited by 7,414
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  12. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
28 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  4. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  5. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  6. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  8. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  9. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  10. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  12. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  13. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  14. 2019Relational Knowledge Distillationcited by 889
  15. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  16. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 807
  17. 2018Born Again Neural Networkscited by 546
  18. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  19. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  20. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  21. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  22. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  23. 2021HOTR: End-to-End Human-Object Interaction Detection with Transformerscited by 183
  24. 2021All Tokens Matter: Token Labeling for Training Better Vision Transformerscited by 144
  25. 2019Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Taskscited by 96
  26. 2021HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrievalcited by 87
  27. 2018Label Refinery: Improving ImageNet Classification through Label Progressioncited by 65
  28. 2022Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuningcited by 33

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-trainingcited by 15
  2. 2025On the Value of Cross-Modal Misalignment in Multimodal Representation Learningcited by 10

Similar, not cited

Close in wording, no citation either way

  1. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining
  2. 2023SILC: Improving Vision Language Pretraining with Self-Distillation
  3. 2021KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillation
  4. 2023DLIP: Distilling Language-Image Pre-training
  5. 2021Contrastive Vision-Language Pre-training with Limited Resources
  6. 2021Data-Efficient Language-Supervised Zero-Shot Learning with Self-Distillation
  7. 2026DINORANKCLIP: DINOv3 Distillation and Injection for Vision-Language Pretraining with High-Order Ranking Consistency
  8. 2022Single-Stream Multi-Level Alignment for Vision-Language Pretraining