mlsanityData to 14 Oct 2026

Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation

arXiv 2203.06386 ↗First posted 12 Mar 2022cs.CL cs.AI cs.CVcited by 70

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

also in 1 more
Abstract (arXiv)

The recent large-scale vision-language pre-training (VLP) of dual-stream architectures (e.g., CLIP) with a tremendous amount of image-text pair data, has shown its superiority on various multimodal alignment tasks. Despite its success, the resulting models are not capable of multimodal generative tasks due to the weak text encoder. To tackle this problem, we propose to augment the dual-stream VLP model with a textual pre-trained language model (PLM) via vision-language knowledge distillation (VLKD), enabling the capability for multimodal generation. VLKD is pretty data- and computation-efficient compared to the pre-training from scratch. Experimental results show that the resulting model has strong zero-shot performance on multimodal generation tasks, such as open-ended visual question answering and image captioning. For example, it achieves 44.5% zero-shot accuracy on the VQAv2 dataset, surpassing the previous state-of-the-art zero-shot model with 7× fewer parameters. Furthermore, the original textual language understanding and generation ability of the PLM is maintained after VLKD, which makes our model versatile for both multimodal and unimodal tasks.

Key facts. Cited by 70 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 60 references that are in the map. First posted on arXiv 12 Mar 2022.

Builds on

48 of 60 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  2. 2021Scaling Up Vision-Language Pre-training for Image Captioning24 Nov
  3. 2021FILIP: Fine-grained Interactive Language-Image Pre-Training9 Nov
  4. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug

Other references · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
32 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  3. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  4. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  5. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  6. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  7. 2019Contrastive Multiview Codingcited by 1,301
  8. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  10. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  11. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  12. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  13. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  14. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  15. 2019Contrastive Representation Distillationcited by 748
  16. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  17. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  18. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  19. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  20. 2016Self-critical Sequence Training for Image Captioningcited by 547
  21. 2018nocaps: novel object captioning at scalecited by 444
  22. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  23. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  24. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
  25. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  26. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  27. 2020GSum: A General Framework for Guided Neural Abstractive Summarizationcited by 129
  28. 2016Guided Open Vocabulary Image Captioning with Constrained Beam Searchcited by 105
  29. 2021M6: A Chinese Multimodal Pretrainercited by 91
  30. 2021MURAL: Multimodal, Multitask Retrieval Across Languagescited by 44
  31. 2021Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarizationcited by 32
  32. 2020Visual Grounding in Video for Unsupervised Word Translationcited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 14; 2023: 28; 2024: 13; 2025: 12; 2026: 3.

70 papers.

Sort:

By area:

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  3. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  4. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  5. 2023Vision-Language Models as Success Detectorscited by 142
  6. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 115
  7. 2022Language Models are General-Purpose Interfacescited by 97
  8. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  9. 2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
  10. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Modelscited by 80
  11. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-trainingcited by 72
  12. 2023Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answeringcited by 51
38 more
  1. 2022MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Promptingcited by 41
  2. 2022NusaCrowd: Open Source Initiative for Indonesian NLP Resourcescited by 38
  3. 2024FROSTER: Frozen CLIP Is A Strong Teacher for Open-Vocabulary Action Recognitioncited by 36
  4. 2023Distilling Large Vision-Language Model with Out-of-Distribution Generalizabilitycited by 33
  5. 2024MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metriccited by 30
  6. 2023Prismer: A Vision-Language Model with Multi-Task Expertscited by 28
  7. 2023Bootstrapping Vision-Language Learning with Decoupled Language Pre-trainingcited by 28
  8. 2023Survey of Social Bias in Vision-Language Modelscited by 26
  9. 2023EdgeFM: Leveraging Foundation Model for Open-set Learning on the Edgecited by 21
  10. 2022FairCLIP: Social Bias Elimination based on Attribute Prototype Learning and Representation Neutralizationcited by 20
  11. 2023CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decodingcited by 20
  12. 2025MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answeringcited by 19
  13. 2024Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalizationcited by 17
  14. 2024PracticalDG: Perturbation Distillation on Vision-Language Models for Hybrid Domain Generalizationcited by 15
  15. 2023MAGVLT: Masked Generative Vision-and-Language Transformercited by 14
  16. 2023Improving Cross-modal Alignment with Synthetic Pairs for Text-only Image Captioningcited by 14
  17. 2024X-Former: Unifying Contrastive and Reconstruction Learning for MLLMscited by 14
  18. 2023CoBIT: A Contrastive Bi-directional Image-Text Generation Modelcited by 11
  19. 2023Zero-shot Visual Question Answering with Language Model Feedbackcited by 11
  20. 2025Multi-modal Knowledge Distillation-based Human Trajectory Forecastingcited by 10
  21. 2023TextManiA: Enriching Visual Feature by Text-driven Manifold Augmentationcited by 9
  22. 2022Zero-shot Image Captioning by Anchor-augmented Vision-Language Space Alignmentcited by 8
  23. 2024Just KIDDIN: Knowledge Infusion and Distillation for Detection of INdecent Memescited by 8
  24. 2022Improving Commonsense in Vision-Language Models via Knowledge Graph Riddlescited by 7
  25. 2023Learning to Imagine: Visually-Augmented Natural Language Generationcited by 7
  26. 2024Towards a text-based quantitative and explainable histopathology image analysiscited by 7
  27. 2025Learning to Generalize without Bias for Open-Vocabulary Action Recognitioncited by 7
  28. 2023ViLTA: Enhancing Vision-Language Pre-training through Textual Augmentationcited by 6
  29. 2023VideoAdviser: Video Knowledge Distillation for Multimodal Transfer Learningcited by 6
  30. 2024MAGIC: Meta-Ability Guided Interactive Chain-of-Distillation for Effective-and-Efficient Vision-and-Language Navigationcited by 6
  31. 2022Gradient Knowledge Distillation for Pre-trained Language Modelscited by 5
  32. 2022Visually-augmented pretrained language models for NLP tasks without imagescited by 5
  33. 2025ToVE: Efficient Vision-Language Learning via Knowledge Transfer from Vision Expertscited by 5
  34. 2023Actional Atomic-Concept Learning for Demystifying Vision-Language Navigationcited by 4
  35. 2023Efficient Cross-Lingual Transfer for Chinese Stable Diffusion with Images as Pivotscited by 4
  36. 2023Lightweight In-Context Tuning for Multimodal Unified Modelscited by 4
  37. 2024Distilling Implicit Multimodal Knowledge into Large Language Models for Zero-Resource Dialogue Generationcited by 4
  38. 2024Training the Untrainable: Introducing Inductive Bias via Representational Alignmentcited by 4

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks
  2. 2023A Survey of Vision-Language Pre-training from the Lens of Multimodal Machine Translation
  3. 2021Unified Multimodal Pre-training and Prompt-based Tuning for Vision-Language Understanding and Generation
  4. 2023DLIP: Distilling Language-Image Pre-training
  5. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trends
  6. 2023MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokens
  7. 2026Let ViT Speak: Generative Language-Image Pre-training
  8. 2024Zoom-shot: Fast and Efficient Unsupervised Zero-Shot Transfer of CLIP to Vision Encoders with Multimodal Loss