mlsanityData to 14 Oct 2026

Lightweight Model Pre-training via Language Guided Knowledge Distillation

arXiv 2406.11689 ↗First posted 17 Jun 2024cs.CVcited by 1

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

This paper studies the problem of pre-training for small models, which is essential for many mobile devices. Current state-of-the-art methods on this problem transfer the representational knowledge of a large network (as a Teacher) into a smaller model (as a Student) using self-supervised distillation, improving the performance of the small model on downstream tasks. However, existing approaches are insufficient in extracting the crucial knowledge that is useful for discerning categories in downstream tasks during the distillation process. In this paper, for the first time, we introduce language guidance to the distillation process and propose a new method named Language-Guided Distillation (LGD) system, which uses category names of the target downstream task to help refine the knowledge transferred between the teacher and student. To this end, we utilize a pre-trained text encoder to extract semantic embeddings from language and construct a textual semantic space called Textual Semantics Bank (TSB). Furthermore, we design a Language-Guided Knowledge Aggregation (LGKA) module to construct the visual semantic space, also named Visual Semantics Bank (VSB). The task-related knowledge is transferred by driving a student encoder to mimic the similarity score distribution inferred by a teacher over TSB and VSB. Compared with other small models obtained by either ImageNet pre-training or self-supervised distillation, experiment results show that the distilled lightweight model using the proposed LGD method presents state-of-the-art performance and is validated on various downstream tasks, including classification, detection, and segmentation. We have made the code available at https://github.com/mZhenz/LGD.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 51 of its 64 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

51 of 64 references.

References · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  9. 2017Mask R-CNNcited by 7,277
  10. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  11. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  12. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
38 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  3. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  4. 2016Pyramid Scene Parsing Networkcited by 2,479
  5. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  6. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,385
  7. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  8. 2021Grounded Language-Image Pre-trainingcited by 1,156
  9. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  10. 2019Contrastive Representation Distillationcited by 748
  11. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  12. 2019Panoptic Feature Pyramid Networkscited by 645
  13. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  14. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  15. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 463
  16. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  17. 2022Decoupled Knowledge Distillationcited by 388
  18. 2022Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Frameworkcited by 379
  19. 2018Rethinking ImageNet Pre-trainingcited by 377
  20. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 320
  21. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 210
  22. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 194
  23. 2022Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoderscited by 144
  24. 2020Cross-Layer Distillation with Semantic Calibrationcited by 143
  25. 2021SEED: Self-supervised Distillation For Visual Representationcited by 134
  26. 2023FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentationcited by 95
  27. 2020CompRess: Self-Supervised Learning by Compressing Representationscited by 62
  28. 2022Dual Modality Prompt Tuning for Vision-Language Pre-Trained Modelcited by 52
  29. 2023Vote2Cap-DETR++: Decoupling Localization and Describing for End-to-End 3D Dense Captioningcited by 39
  30. 2023M3DBench: Let's Instruct Large Models with Multi-modal 3D Promptscited by 32
  31. 2021TCGL: Temporal Contrastive Graph for Self-supervised Video Representation Learningcited by 31
  32. 2021DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learningcited by 26
  33. 2021Bag of Instances Aggregation Boosts Self-supervised Distillationcited by 22
  34. 2022Knowledge Distillation as Efficient Pre-training: Faster Convergence, Higher Data-efficiency, and Better Transferabilitycited by 21
  35. 2021On the Efficacy of Small Self-Supervised Contrastive Models without Distillation Signalscited by 14
  36. 2021Self-Supervised Learning by Estimating Twin Class Distributionscited by 12
  37. 2023Effective End-to-End Vision Language Pretraining with Semantic Visual Losscited by 10
  38. 2022Beyond Instance Discrimination: Relation-aware Contrastive Self-supervised Learningcited by 4

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025MOCHA: Multi-modal Objects-aware Cross-arcHitecture Alignmentcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Efficient Knowledge Distillation: Empowering Small Language Models with Teacher Model Insights
  2. 2026KDFP: A first-principles approach to knowledge distillation in large language models
  3. 2022Gradient Knowledge Distillation for Pre-trained Language Models
  4. 2023A Cohesive Distillation Architecture for Neural Language Models
  5. 2026Large Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledge
  6. 2022Multi-level Distillation of Semantic Knowledge for Pre-training Multilingual Language Model
  7. 2024Pre-training Distillation for Large Language Models: A Design Space Exploration
  8. 2026From Images to Words: Efficient Cross-Modal Knowledge Distillation to Language Models from Black-box Teachers