mlsanityData to 14 Oct 2026

DLIP: Distilling Language-Image Pre-training

arXiv 2308.12956 ↗First posted 24 Aug 2023cs.CV cs.AI cs.LGcited by 5

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

also in 1 more
Abstract (arXiv)

Vision-Language Pre-training (VLP) shows remarkable progress with the assistance of extremely heavy parameters, which challenges deployment in real applications. Knowledge distillation is well recognized as the essential procedure in model compression. However, existing knowledge distillation techniques lack an in-depth investigation and analysis of VLP, and practical guidelines for VLP-oriented distillation are still not yet explored. In this paper, we present DLIP, a simple yet efficient Distilling Language-Image Pre-training framework, through which we investigate how to distill a light VLP model. Specifically, we dissect the model distillation from multiple dimensions, such as the architecture characteristics of different modules and the information transfer of different modalities. We conduct comprehensive experiments and provide insights on distilling a light but performant VLP model. Experimental results reveal that DLIP can achieve a state-of-the-art accuracy/efficiency trade-off across diverse cross-modal tasks, e.g., image-text retrieval, image captioning and visual question answering. For example, DLIP compresses BLIP by 1.9x, from 213M to 108M parameters, while achieving comparable or better performance. Furthermore, DLIP succeeds in retaining more than 95% of the performance with 22.4% parameters and 24.8% FLOPs compared to the teacher model and accelerates inference speed by 2.7x.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 58 references that are in the map. First posted on arXiv 24 Aug 2023.

Builds on

51 of 58 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  2. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug

Other references · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2023GPT-4 Technical Reportcited by 18,396
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  8. 2022Training language models to follow instructions with human feedbackcited by 15,971
  9. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  10. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  11. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  12. 2023Visual Instruction Tuningcited by 7,869
37 more
  1. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  6. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  8. 2015VQA: Visual Question Answeringcited by 2,506
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  10. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  11. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  12. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  13. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  14. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  15. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  16. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  17. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  18. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  19. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  20. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  21. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  22. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  23. 2019Relational Knowledge Distillationcited by 889
  24. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  25. 2023Instruction Tuning with GPT-4cited by 686
  26. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  27. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  28. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  29. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  30. 2018Model compression via distillation and quantizationcited by 289
  31. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  32. 2016Image Captioning with Semantic Attentioncited by 260
  33. 2021Knowledge distillation: A good teacher is patient and consistentcited by 259
  34. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 86
  35. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
  36. 2020MiniVLM: A Smaller and Faster Vision-Language Modelcited by 51
  37. 2022ViTKD: Practical Guidelines for ViT feature knowledge distillationcited by 34

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 1; 2026: 1.

5 papers.

  1. 2023MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Trainingcited by 96
  2. 2025MobileCLIP2: Improving Multi-Modal Reinforced Trainingcited by 21
  3. 2024MAGIC: Meta-Ability Guided Interactive Chain-of-Distillation for Effective-and-Efficient Vision-and-Language Navigationcited by 6
  4. 2024Zoom-shot: Fast and Efficient Unsupervised Zero-Shot Transfer of CLIP to Vision Encoders with Multimodal Losscited by 3
  5. 2026PVCap: Towards Accurate 3D Dense Captioning via PseudoCap and VoxelCapNetcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trends
  2. 2022VLP: A Survey on Vision-Language Pre-training
  3. 2021KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillation
  4. 2022Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation
  5. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks
  6. 2022Compressing And Debiasing Vision-Language Pre-Trained Models for Visual Question Answering
  7. 2026GoldiCLIP: The Goldilocks Approach for Balancing Explicit Supervision for Language-Image Pretraining
  8. 2025When Better Teachers Don't Make Better Students: Revisiting Knowledge Distillation for CLIP Models in VQA