mlsanityData to 14 Oct 2026

DisCo: Distilled Student Models Co-training for Semi-supervised Text Mining

arXiv 2305.12074 ↗First posted 20 May 2023cs.CLcited by 4

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Many text mining models are constructed by fine-tuning a large deep pre-trained language model (PLM) in downstream tasks. However, a significant challenge nowadays is maintaining performance when we use a lightweight model with limited labelled samples. We present DisCo, a semi-supervised learning (SSL) framework for fine-tuning a cohort of small student models generated from a large PLM using knowledge distillation. Our key insight is to share complementary knowledge among distilled student cohorts to promote their SSL effectiveness. DisCo employs a novel co-training technique to optimize a cohort of multiple small student models by promoting knowledge sharing among students under diversified views: model views produced by different distillation strategies and data views produced by various input augmentations. We evaluate DisCo on both semi-supervised text classification and extractive summarization tasks. Experimental results show that DisCo can produce student models that are 7.6 times smaller and 4.8 times faster in inference than the baseline PLMs while maintaining comparable performance. We also show that DisCo-generated student models outperform the similar-sized models elaborately tuned in distinct tasks.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 49 of its 71 references that are in the map. First posted on arXiv 20 May 2023.

Builds on

49 of 71 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2016Pointer Sentinel Mixture Modelscited by 3,096
  9. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  10. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  11. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,085
  12. 2016Adversarial examples in the physical worldcited by 1,769
37 more
  1. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  2. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  3. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,391
  4. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  5. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  6. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,127
  7. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,060
  8. 2015Teaching Machines to Read and Comprehendcited by 1,049
  9. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 920
  10. 2017Deep Mutual Learningcited by 628
  11. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  12. 2019Text Summarization with Pretrained Encoderscited by 556
  13. 2018Born Again Neural Networkscited by 546
  14. 2021FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labelingcited by 540
  15. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  16. 2013Do Deep Nets Really Need to be Deep?cited by 474
  17. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 464
  18. 2016Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learningcited by 441
  19. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 359
  20. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 254
  21. 2015Semi-Supervised Learning with Ladder Networkscited by 247
  22. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  23. 2020MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classificationcited by 187
  24. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  25. 2018Deep Co-Training for Semi-Supervised Image Recognitioncited by 147
  26. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 134
  27. 2022SimMatch: Semi-supervised Learning with Similarity Matchingcited by 118
  28. 2021AdaMatch: A Unified Approach to Semi-Supervised Learning and Domain Adaptationcited by 104
  29. 2022USB: A Unified Semi-supervised Learning Benchmark for Classificationcited by 95
  30. 2020Densely Guided Knowledge Distillation using Multiple Teacher Assistantscited by 83
  31. 2020A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generationcited by 72
  32. 2020EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inferencecited by 70
  33. 2019Variational Pretraining for Semi-supervised Text Classificationcited by 41
  34. 2021Revisiting Consistency Regularization for Semi-Supervised Learningcited by 34
  35. 2020SLM: Learning a Discourse Language Representation with Sentence Unshufflingcited by 24
  36. 2022Progressive Class Semantic Matching for Semi-supervised Text Classificationcited by 7
  37. 2021FLiText: A Faster and Lighter Semi-Supervised Text Classification with Convolution Networkscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2024Lightweight Contenders: Navigating Semi-Supervised Text Mining through Peer Collaboration and Self Transcendencecited by 1
  2. 2025Enhancing Semi-supervised Learning with Zero-shot Pseudolabelscited by 1
  3. 2025Conceptual Contrastive Edits in Textual and Vision-Language Retrievalcited by 0
  4. 2026NanoNet: Parameter-Efficient Learning with Label-Scarce Supervision for Lightweight Text Mining Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LLM on a Budget: Active Knowledge Distillation for Efficient Classification of Large Text Corpora
  2. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Models
  3. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes
  4. 2023KDSTM: Neural Semi-supervised Topic Modeling with Knowledge Distillation
  5. 2018Mixture of Expert/Imitator Networks: Scalable Semi-supervised Learning Framework
  6. 2021Sparse Distillation: Speeding Up Text Classification by Using Bigger Student Models
  7. 2023Multistage Collaborative Knowledge Distillation from a Large Language Model for Semi-Supervised Sequence Generation
  8. 2024Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labels