mlsanityData to 14 Oct 2026

KDSTM: Neural Semi-supervised Topic Modeling with Knowledge Distillation

arXiv 2307.01878 ↗First posted 4 Jul 2023cs.CL cs.AIcited by 3

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

In text classification tasks, fine tuning pretrained language models like BERT and GPT-3 yields competitive accuracy; however, both methods require pretraining on large text datasets. In contrast, general topic modeling methods possess the advantage of analyzing documents to extract meaningful patterns of words without the need of pretraining. To leverage topic modeling's unsupervised insights extraction on text classification tasks, we develop the Knowledge Distillation Semi-supervised Topic Modeling (KDSTM). KDSTM requires no pretrained embeddings, few labeled documents and is efficient to train, making it ideal under resource constrained settings. Across a variety of datasets, our method outperforms existing supervised topic modeling methods in classification accuracy, robustness and efficiency and achieves similar performance compare to state of the art weakly supervised text classification methods.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 32 of its 60 references that are in the map. First posted on arXiv 4 Jul 2023.

Builds on

32 of 60 references.

References · 32

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  5. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  7. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  8. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  9. 2019Similarity-Preserving Knowledge Distillationcited by 516
  10. 2014Semi-Supervised Learning with Deep Generative Modelscited by 496
  11. 2018Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Networkcited by 420
  12. 2019Topic Modeling in Embedding Spacescited by 224
20 more
  1. 2018Hyperspherical Variational Auto-Encoderscited by 189
  2. 2020Cross-Layer Distillation with Semantic Calibrationcited by 143
  3. 2020Multilingual Alignment of Contextual Word Representationscited by 114
  4. 2017Latent Dirichlet Allocation (LDA) and Topic modeling: models, applications, a surveycited by 91
  5. 2017Generating Sentences by Editing Prototypescited by 88
  6. 2017Discovering Discrete Latent Topics with Neural Variational Inferencecited by 79
  7. 2018Spherical Latent Spaces for Stable Variational Autoencoderscited by 67
  8. 2020ALP-KD: Attention-Based Layer Projection for Knowledge Distillationcited by 66
  9. 2017DarkRank: Accelerating Deep Metric Learning via Cross Sample Similarities Transfercited by 56
  10. 2020Neural Topic Model via Optimal Transportcited by 50
  11. 2018Weakly-Supervised Hierarchical Text Classificationcited by 42
  12. 2021A Survey on Optimal Transport for Machine Learning: Theory and Applicationscited by 35
  13. 2019Improving Sequence-to-Sequence Learning via Optimal Transportcited by 33
  14. 2016TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependencycited by 27
  15. 2020Improving Neural Topic Models using Knowledge Distillationcited by 25
  16. 2018Distilled Wasserstein Learning for Word Embedding and Topic Modelingcited by 18
  17. 2016Nonparametric Spherical Topic Modeling with Word Embeddingscited by 16
  18. 2023vONTSS: vMF based semi-supervised neural topic modeling with optimal transportcited by 16
  19. 2017Identifying Patterns of Associated-Conditions through Topic Models of Electronic Medical Recordscited by 5
  20. 2015End-to-end Learning of LDA by Mirror-Descent Back Propagation over a Deep Architecturecited by 2

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 1.

3 papers.

  1. 2025Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Modelscited by 6
  2. 2023S2vNTM: Semi-supervised vMF Neural Topic Modelingcited by 1
  3. 2024JADS: A Framework for Self-supervised Joint Aspect Discovery and Summarizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Improving Neural Topic Models with Wasserstein Knowledge Distillation
  2. 2022Which Student is Best? A Comprehensive Knowledge Distillation Exam for Task-Specific BERT Models
  3. 2023DisCo: Distilled Student Models Co-training for Semi-supervised Text Mining
  4. 2024Performance-Guided LLM Knowledge Distillation for Efficient Text Classification at Scale
  5. 2026NanoNet: Parameter-Efficient Learning with Label-Scarce Supervision for Lightweight Text Mining Model
  6. 2024Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labels
  7. 2024Intermediate Distillation: Data-Efficient Distillation from Black-Box LLMs for Information Retrieval
  8. 2024LLM Teacher-Student Framework for Text Classification With No Manually Annotated Data: A Case Study in IPTC News Topic Classification