mlsanityData to 14 Oct 2026

torchdistill Meets Hugging Face Libraries for Reproducible, Coding-Free Deep Learning Studies: A Case Study on NLP

arXiv 2310.17644 ↗First posted 26 Oct 2023cs.CL cs.CV cs.LGcited by 2

On the map: Knowledge distillation → Multimodal distillation

Abstract (arXiv)

Reproducibility in scientific work has been becoming increasingly important in research communities such as machine learning, natural language processing, and computer vision communities due to the rapid development of the research domains supported by recent advances in deep learning. In this work, we present a significantly upgraded version of torchdistill, a modular-driven coding-free deep learning framework significantly upgraded from the initial release, which supports only image classification and object detection tasks for reproducible knowledge distillation experiments. To demonstrate that the upgraded framework can support more tasks with third-party libraries, we reproduce the GLUE benchmark results of BERT models using a script based on the upgraded torchdistill, harmonizing with various Hugging Face libraries. All the 27 fine-tuned BERT models and configurations to reproduce the results are published at Hugging Face, and the model weights have already been widely used in research communities. We also reimplement popular small-sized models and new knowledge distillation methods and perform additional experiments for computer vision tasks.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Multimodal distillation, one of 12 directions in Knowledge distillation. Builds on 54 of its 86 references that are in the map. First posted on arXiv 26 Oct 2023.

Builds on

54 of 86 references.

References · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  8. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  9. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  10. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  11. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  12. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
38 more
  1. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  2. 2016Densely Connected Convolutional Networkscited by 6,720
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  7. 2013Auto-Encoding Variational Bayescited by 4,053
  8. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  9. 2016Wide Residual Networkscited by 3,070
  10. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  12. 2017Billion-scale similarity search with GPUscited by 2,497
  13. 2019Searching for MobileNetV3cited by 1,994
  14. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  15. 2018Neural Network Acceptability Judgmentscited by 958
  16. 2019Relational Knowledge Distillationcited by 889
  17. 2019Contrastive Representation Distillationcited by 748
  18. 2016End-to-end Optimized Image Compressioncited by 696
  19. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  20. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  21. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  22. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  23. 2018AllenNLP: A Deep Semantic Natural Language Processing Platformcited by 347
  24. 2020Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)cited by 322
  25. 2021Distilling Knowledge via Knowledge Reviewcited by 288
  26. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 260
  27. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 238
  28. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 213
  29. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 169
  30. 2020Knowledge Distillation Meets Self-Supervisioncited by 142
  31. 2019Knowledge Adaptation for Efficient Semantic Segmentationcited by 105
  32. 2022ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantizationcited by 51
  33. 2021Just What do You Think You're Doing, Dave?' A Checklist for Responsible Data Use in NLPcited by 47
  34. 2022Exploring Inter-Channel Correlation for Diversity-preserved KnowledgeDistillationcited by 44
  35. 2023Understanding the Role of the Projector in Knowledge Distillationcited by 31
  36. 2020Prime-Aware Adaptive Distillationcited by 28
  37. 2022BottleFit: Learning Compressed Representations in Deep Neural Networks for Effective and Efficient Split Computingcited by 28
  38. 2019Critically Examining the "Neural Hype": Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Modelscited by 24

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 1.

2 papers.

  1. 2024Exploring the Use of Abusive Generative AI Models on Civitaicited by 10
  2. 2026SecureRouter: Encrypted Routing for Efficient Secure Inferencecited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024AdaDistill: Adaptive Knowledge Distillation for Deep Face Recognition
  2. 2026DVTR-KD: Distilling Vision-Language Models via Intermediate Visual Token Reconstruction
  3. 2026DataComp-VLM: Improved Open Datasets for Vision-Language Models
  4. 2023SynthDistill: Face Recognition with Knowledge Distillation from Synthetic Data
  5. 2024Distilling Vision-Language Foundation Models: A Data-Free Approach via Prompt Diversification
  6. 2025GreedyPrune: Retenting Critical Visual Token Set for Large Vision Language Models
  7. 2025RedVTP: Training-Free Acceleration of Diffusion Vision-Language Models Inference via Masked Token-Guided Visual Token Pruning
  8. 2025Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimization