mlsanityData to 7 Oct 2026

XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation

arXiv 2106.04563 ↗First posted 8 Jun 2021cs.CL cs.AI cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

While deep and large pre-trained models are the state-of-the-art for various natural language processing tasks, their huge size poses significant challenges for practical uses in resource constrained settings. Recent works in knowledge distillation propose task-agnostic as well as task-specific methods to compress these models, with task-specific ones often yielding higher compression rate. In this work, we develop a new task-agnostic distillation framework XtremeDistilTransformers that leverages the advantage of task-specific methods for learning a small universal model that can be applied to arbitrary tasks and languages. To this end, we study the transferability of several source tasks, augmentation resources and model architecture for distillation. We evaluate our model performance on multiple tasks, including the General Language Understanding Evaluation (GLUE) benchmark, SQuAD question answering dataset and a massive multi-lingual NER dataset with 41 languages. We release three distilled task-agnostic checkpoints with 13MM, 22MM and 33MM parameters obtaining SOTA performance in several tasks.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 31 of its 40 references that are in the map. First posted on arXiv 8 Jun 2021.

Builds on

31 of 40 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  11. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  12. 2015A large annotated corpus for learning natural language inferencecited by 1,837
19 more
  1. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  2. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  3. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  4. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  5. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  6. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  7. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  8. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  9. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  10. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 189
  11. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  12. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 152
  13. 2020Self-training Improves Pre-training for Natural Language Understandingcited by 96
  14. 2019Knowledge Distillation from Internal Representationscited by 87
  15. 2017ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translationscited by 44
  16. 2020XtremeDistil: Multi-stage Distillation for Massive Multilingual Modelscited by 38
  17. 2019Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understandingcited by 37
  18. 2020LRC-BERT: Latent-representation Contrastive Knowledge Distillation for Natural Language Understandingcited by 27
  19. 2019Extremely Small BERT Models from Mixed-Vocabulary Trainingcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 9; 2024: 2; 2025: 2; 2026: 1.

18 papers.

Sort:

By area:

  1. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  2. 2021MetaQA: Combining Expert Agents for Multi-Skill Question Answeringcited by 15
  3. 2023Better Language Models of Code through Self-Improvementcited by 14
  4. 2023Neural Architecture Search for Effective Teacher-Student Knowledge Transfer in Language Modelscited by 6
  5. 2023Distilling Efficient Language-Specific Models for Cross-Lingual Transfercited by 6
  6. 2023A Comparative Analysis of Task-Agnostic Distillation Methods for Compressing Transformer Language Modelscited by 6
  7. 2023From Big to Small Without Losing It All: Text Augmentation with ChatGPT for Efficient Sentiment Analysiscited by 6
  8. 2025Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Modelscited by 6
  9. 2022Too Brittle To Touch: Comparing the Stability of Quantization and Distillation Towards Developing Lightweight Low-Resource MT Modelscited by 5
  10. 2024Efficiently Distilling LLMs for Edge Applicationscited by 5
  11. 2023Just CHOP: Embarrassingly Simple LLM Compressioncited by 4
  12. 2023Exploring the Performance and Efficiency of Transformer Models for NLP on Mobile Devicescited by 4
6 more
  1. 2023Exponentially Faster Language Modellingcited by 4
  2. 2023ERNIE 3.0 Tiny: Frustratingly Simple Method to Improve Task-Agnostic Distillation Generalizationcited by 3
  3. 2025Contrastive Knowledge Transfer and Robust Optimization for Secure Alignment of Large Language Modelscited by 3
  4. 2022Compressing Cross-Lingual Multi-Task Models at Qualtricscited by 1
  5. 2024CARIn: Constraint-Aware and Responsive Inference on Heterogeneous Devices for Single- and Multi-DNN Workloadscited by 1
  6. 2026MANAR: Memory-augmented Attention with Navigational Abstract Conceptual Representationcited by 0