mlsanityData to 7 Oct 2026

Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inference

arXiv 2306.02379 ↗First posted 4 Jun 2023cs.CL cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Pre-trained Transformer models like T5 and BART have advanced the state of the art on a wide range of text generation tasks. Compressing these models into smaller ones has become critically important for practical use. Common neural network compression techniques such as knowledge distillation or quantization are limited to static compression where the compression ratio is fixed. In this paper, we introduce Modular Transformers, a modularized encoder-decoder framework for flexible sequence-to-sequence model compression. Modular Transformers train modularized layers that have the same function of two or more consecutive layers in the original model via module replacing and knowledge distillation. After training, the modularized layers can be flexibly assembled into sequence-to-sequence models that meet different performance-efficiency trade-offs. Experimental results show that after a single training phase, by simply varying the assembling strategy, Modular Transformers can achieve flexible compression ratios from 1.1x to 6x with little to moderate relative performance drop.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 44 of its 54 references that are in the map. First posted on arXiv 4 Jun 2023.

Builds on

44 of 54 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  10. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  11. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  12. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
32 more
  1. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  2. 2015Teaching Machines to Read and Comprehendcited by 1,039
  3. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  4. 2019Are Sixteen Heads Really Better than One?cited by 803
  5. 2016Sequence-Level Knowledge Distillationcited by 761
  6. 2019Green AIcited by 620
  7. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  8. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  9. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  10. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  11. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  12. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  13. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  14. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  15. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  16. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  17. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  18. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125
  19. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 112
  20. 2020Pre-trained Summarization Distillationcited by 70
  21. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
  22. 2019Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translationcited by 56
  23. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
  24. 2021A Survey on Green Deep Learningcited by 52
  25. 2018Harvesting Paragraph-Level Question-Answer Pairs from Wikipediacited by 49
  26. 2021Differentiable Subset Pruning of Transformer Headscited by 46
  27. 2021Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compressioncited by 38
  28. 2022DQ-BART: Efficient Sequence-to-Sequence Model via Joint Distillation and Quantizationcited by 23
  29. 2023Efficient Prompting via Dynamic In-Context Learningcited by 18
  30. 2021Attention Temperature Matters in Abstractive Summarization Distillationcited by 14
  31. 2021Improving Sequence-to-Sequence Pre-training via Sequence Span Rewritingcited by 13
  32. 2021DACT-BERT: Differentiable Adaptive Computation Time for an Efficient BERT Inferencecited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Efficient Agents: Building Effective Agents While Reducing Costcited by 32
  2. 2025UORA: Uniform Orthogonal Reinitialization Adaptation in Parameter-Efficient Fine-Tuning of Large Modelscited by 11
  3. 2026LoopQ: Quantization for Recursive Transformerscited by 4
  4. 2026Locking Pretrained Weights via Deep Low-Rank Residual Distillationcited by 0