mlsanityData to 7 Oct 2026

A Short Study on Compressing Decoder-Based Language Models

arXiv 2110.08460 ↗First posted 16 Oct 2021cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

Pre-trained Language Models (PLMs) have been successful for a wide range of natural language processing (NLP) tasks. The state-of-the-art of PLMs, however, are extremely large to be used on edge devices. As a result, the topic of model compression has attracted increasing attention in the NLP community. Most of the existing works focus on compressing encoder-based models (tiny-BERT, distilBERT, distilRoBERTa, etc), however, to the best of our knowledge, the compression of decoder-based models (such as GPT-2) has not been investigated much. Our paper aims to fill this gap. Specifically, we explore two directions: 1) we employ current state-of-the-art knowledge distillation techniques to improve fine-tuning of DistilGPT-2. 2) we pre-train a compressed GPT-2 model using layer truncation and compare it against the distillation-based method (DistilGPT2). The training time of our compressed model is significantly less than DistilGPT-2, but it can achieve better performance when fine-tuned on downstream tasks. We also demonstrate the impact of data cleaning on model performance.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 28 of its 40 references that are in the map. First posted on arXiv 16 Oct 2021.

Builds on

28 of 40 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  10. 2018Deep contextualized word representationscited by 2,285
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
16 more
  1. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  2. 2018Born Again Neural Networkscited by 539
  3. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  4. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 341
  5. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  6. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  7. 2020Regularizing Class-wise Predictions via Self-knowledge Distillationcited by 144
  8. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 112
  9. 2020ALP-KD: Attention-Based Layer Projection for Knowledge Distillationcited by 65
  10. 2020Compression of Deep Learning Models for Text: A Surveycited by 64
  11. 2021Annealing Knowledge Distillationcited by 61
  12. 2021MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillationcited by 27
  13. 2020Towards Zero-Shot Knowledge Distillation for Natural Language Processingcited by 20
  14. 2021RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillationcited by 18
  15. 2021Not Far Away, Not So Close: Sample Efficient Nearest Neighbour Data Augmentation via MiniMaxcited by 10
  16. 2021How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understandingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 7; 2024: 5; 2025: 2; 2026: 1.

16 papers.

Sort:

  1. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  2. 2024A Survey on Transformer Compressioncited by 44
  3. 2022In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Modelscited by 36
  4. 2023What Matters In The Structured Pruning of Generative Language Models?cited by 30
  5. 2023Training Acceleration of Low-Rank Decomposed Networks using Sequential Freezing and Rank Quantizationcited by 8
  6. 2023Speeding up Resnet Architecture with Layers Targeted Low Rank Decompositioncited by 8
  7. 2024SkipViT: Speeding Up Vision Transformers with a Token-Level Skip Connectioncited by 5
  8. 2023IDTraffickers: An Authorship Attribution Dataset to link and connect Potential Human-Trafficking Operations on Text Escort Advertisementscited by 3
  9. 2023SwiftLearn: A Data-Efficient Training Method of Deep Learning Models using Importance Samplingcited by 3
  10. 2024Merino: Entropy-driven Design for Generative Language Models on IoT Devicescited by 3
  11. 2025Domain-Adaptive Small Language Models for Structured Tax Code Predictioncited by 3
  12. 2023Learn What Is Possible, Then Choose What Is Best: Disentangling One-To-Many Relations in Language Through Text-based Gamescited by 2
4 more
  1. 2023Improving Resnet-9 Generalization Trained on Small Datasetscited by 2
  2. 2024Generating Accurate Synthetic Survival Data by Conditioning on Outcomescited by 2
  3. 2025An Empirical Study of Knowledge Distillation for Code Understanding Taskscited by 2
  4. 2026Model Multiplicity for Adversarial Detection in Small Language Model Training on Edge Devicescited by 0