# Transformer to CNN: Label-scarce distillation for efficient text classification

arXiv: https://arxiv.org/abs/1909.03508 · HTML: https://mlsanity.com/paper/1909.03508/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Sep 2019.

## Built on by (most cited first, 9 of 9)

- [Distilling Linguistic Context for Language Model Compression](https://mlsanity.com/paper/2109.08359/) (2021)
- [Knowledge Distillation of Russian Language Models with Reduction of Vocabulary](https://mlsanity.com/paper/2205.02340/) (2022)
- [TransMamba: Fast Universal Architecture Adaption from Transformers to Mamba](https://mlsanity.com/paper/2502.15130/) (2025)
- [DistillCSE: Distilled Contrastive Learning for Sentence Embeddings](https://mlsanity.com/paper/2310.13499/) (2023)
- [Low-Latency Incremental Text-to-Speech Synthesis with Distilled Context Prediction Network](https://mlsanity.com/paper/2109.10724/) (2021)
- [Sparse Distillation: Speeding Up Text Classification by Using Bigger Student Models](https://mlsanity.com/paper/2110.08536/) (2021)
- [RefBERT: Compressing BERT by Referencing to Pre-computed Representations](https://mlsanity.com/paper/2106.08898/) (2021)
- [Can depth-adaptive BERT perform better on binary classification tasks](https://mlsanity.com/paper/2111.10951/) (2021)
- [VLCQ: Post-training quantization for deep neural networks using variable length coding](https://mlsanity.com/paper/2610.15092/) (2026)

## Abstract (arXiv)

Significant advances have been made in Natural Language Processing (NLP) modelling since the beginning of 2018. The new approaches allow for accurate results, even when there is little labelled data, because these NLP models can benefit from training on both task-agnostic and task-specific unlabelled data. However, these advantages come with significant size and computational costs. This workshop paper outlines how our proposed convolutional student architecture, having been trained by a distillation process from a large-scale model, can achieve 300x inference speedup and 39x reduction in parameter count. In some cases, the student model performance surpasses its teacher on the studied tasks.