mlsanityData to 7 Oct 2026

BERT-of-Theseus: Compressing BERT by Progressive Module Replacing

arXiv 2002.02925 ↗First posted 7 Feb 2020cs.CL cs.LGcited by 112

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a novel model compression approach to effectively compress BERT by progressive module replacing. Our approach first divides the original BERT into several modules and builds their compact substitutes. Then, we randomly replace the original modules with their substitutes to train the compact modules to mimic the behavior of the original modules. We progressively increase the probability of replacement through the training. In this way, our approach brings a deeper level of interaction between the original and compact models. Compared to the previous knowledge distillation approaches for BERT compression, our approach does not introduce any additional loss function. Our approach outperforms existing knowledge distillation approaches on GLUE benchmark, showing a new perspective of model compression.

Key facts. Cited by 112 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 32; 2022: 27; 2023: 22; 2024: 9; 2025: 17; 2026: 5.

112 papers.

Sort:

By area:

  1. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  2. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 103
  3. 2023Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generationcited by 94
  4. 2022Less is More: Task-aware Layer-wise Distillation for Language Model Compressioncited by 91
  5. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 85
  6. 2022Unified Visual Transformer Compressioncited by 79
  7. 2022Representation Compensation Networks for Continual Semantic Segmentationcited by 72
  8. 2023Inference with Reference: Lossless Acceleration of Large Language Modelscited by 63
  9. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
  10. 2021Compute and Energy Consumption Trends in Deep Learning Inferencecited by 61
  11. 2021Go Wider Instead of Deepercited by 56
  12. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
38 more
  1. 2021NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Searchcited by 55
  2. 2021A Survey on Green Deep Learningcited by 52
  3. 2022Parameter-Efficient and Student-Friendly Knowledge Distillationcited by 47
  4. 2023ZipLM: Inference-Aware Structured Pruning of Language Modelscited by 46
  5. 2021Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinesecited by 45
  6. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  7. 2021Towards Efficient NLP: A Standard Evaluation and A Strong Baselinecited by 44
  8. 2024A Survey on Transformer Compressioncited by 44
  9. 2021Generate, Annotate, and Learn: NLP with Synthetic Textcited by 42
  10. 2024Everybody Prune Now: Structured Pruning of LLMs with only Forward Passescited by 42
  11. 2021Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigmcited by 40
  12. 2022Wavelet Knowledge Distillation: Towards Efficient Image-to-Image Translationcited by 39
  13. 2021Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compressioncited by 38
  14. 2021Dynamic Knowledge Distillation for Pre-trained Language Modelscited by 38
  15. 2022MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptationcited by 37
  16. 2022In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Modelscited by 36
  17. 2021NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Frameworkcited by 34
  18. 2021Distilling Linguistic Context for Language Model Compressioncited by 33
  19. 2022PointDistiller: Structured Knowledge Distillation Towards Efficient and Compact 3D Detectioncited by 33
  20. 2022A Simple Hash-Based Early Exiting Approach For Language Understanding and Generationcited by 32
  21. 2025Efficient Agents: Building Effective Agents While Reducing Costcited by 32
  22. 2023Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Modelscited by 30
  23. 2023HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformerscited by 25
  24. 2024Pre-training Distillation for Large Language Models: A Design Space Explorationcited by 25
  25. 2022A Survey on Dynamic Neural Networks for Natural Language Processingcited by 24
  26. 2022Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-trainingcited by 24
  27. 2023Grounding Foundation Models through Federated Transfer Learning: A General Frameworkcited by 24
  28. 2022Multimodal Adaptive Distillation for Leveraging Unimodal Encoders for Vision-Language Taskscited by 22
  29. 2022EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generationcited by 21
  30. 2021Analyzing and Mitigating Interference in Neural Architecture Searchcited by 20
  31. 2021Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigmcited by 20
  32. 2021Early Exiting with Ensemble Internal Classifierscited by 19
  33. 2021KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillationcited by 19
  34. 2022AdapLeR: Speeding up Inference by Adaptive Length Reductioncited by 19
  35. 2023Efficient Prompting via Dynamic In-Context Learningcited by 18
  36. 2023F-PABEE: Flexible-patience-based Early Exiting for Single-label and Multi-label text Classification Taskscited by 18
  37. 2021NewsBERT: Distilling Pre-trained Language Model for Intelligent News Applicationcited by 17
  38. 2022Write and Paint: Generative Vision-Language Models are Unified Modal Learnerscited by 17

50 most cited of 112 All 112 (JSON)