mlsanityData to 7 Oct 2026

RomeBERT: Robust Training of Multi-Exit BERT

arXiv 2101.09755 ↗First posted 24 Jan 2021cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

BERT has achieved superior performances on Natural Language Understanding (NLU) tasks. However, BERT possesses a large number of parameters and demands certain resources to deploy. For acceleration, Dynamic Early Exiting for BERT (DeeBERT) has been proposed recently, which incorporates multiple exits and adopts a dynamic early-exit mechanism to ensure efficient inference. While obtaining an efficiency-performance tradeoff, the performances of early exits in multi-exit BERT are significantly worse than late exits. In this paper, we leverage gradient regularized self-distillation for RObust training of Multi-Exit BERT (RomeBERT), which can effectively solve the performance imbalance problem between early and late exits. Moreover, the proposed RomeBERT adopts a one-stage joint training strategy for multi-exits and the BERT backbone while DeeBERT needs two stages that require more training time. Extensive experiments on GLUE datasets are performed to demonstrate the superiority of our approach. Our code is available at https://github.com/romebert/RomeBERT.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 26 of its 28 references that are in the map. First posted on arXiv 24 Jan 2021.

Builds on

26 of 28 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2017Non-local Neural Networkscited by 2,290
  7. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  8. 2020Reformer: The Efficient Transformercited by 1,658
  9. 2020Rethinking Attention with Performerscited by 1,584
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  12. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
14 more
  1. 2016Deep Networks with Stochastic Depthcited by 973
  2. 2018Born Again Neural Networkscited by 539
  3. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  4. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  5. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  6. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  7. 2021Fast Convergence of DETR with Spatially Modulated Co-Attentioncited by 176
  8. 2020Fast Transformers with Clustered Attentioncited by 123
  9. 2016Spatially Adaptive Computation Time for Residual Networkscited by 118
  10. 2020End-to-End Object Detection with Adaptive Clustering Transformercited by 109
  11. 2018Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Studentscited by 50
  12. 2020Dynamic Graph Representation Learning for Video Dialog via Multi-Modal Shuffled Transformerscited by 7
  13. 2020Multi-Pass Transformer for Machine Translationcited by 5
  14. 2020Character Matters: Video Story Understanding with Character-Aware Relationscited by 4

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 3; 2023: 6; 2024: 5; 2025: 2.

19 papers.

Sort:

By area:

  1. 2023AutoMix: Automatically Mixing Language Modelscited by 116
  2. 2023Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMscited by 111
  3. 2021Consistent Accelerated Inference via Confident Adaptive Transformerscited by 73
  4. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  5. 2021A Survey on Green Deep Learningcited by 52
  6. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  7. 2023Uncertainty in Natural Language Processing: Sources, Quantification, and Applicationscited by 44
  8. 2024Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similaritycited by 36
  9. 2022A Survey on Dynamic Neural Networks for Natural Language Processingcited by 24
  10. 2024Agreement-Based Cascading for Efficient Inferencecited by 16
  11. 2023ConsistentEE: A Consistent and Hardness-Guided Early Exiting Method for Accelerating Language Models Inferencecited by 13
  12. 2023DEED: Dynamic Early Exit on Decoder for Accelerating Encoder-Decoder Transformer Modelscited by 7
7 more
  1. 2025BEEM: Boosting Performance of Early Exit DNNs using Multi-Exit Classifiers as Expertscited by 6
  2. 2023Training dynamic models using early exits for automatic speech recognition on resource-constrained devicescited by 3
  3. 2021Scalable Transformers for Neural Machine Translationcited by 2
  4. 2022FLOWGEN: Fast and slow graph generationcited by 2
  5. 2024Multimodal Adaptive Inference for Document Image Classification with Anytime Early Exitingcited by 2
  6. 2024Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Modelscited by 0
  7. 2025Two-Stage Voting for Robust and Efficient Suicide Risk Detection on Social Mediacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023SmartBERT: A Promotion of Dynamic Early Exiting Mechanism for Accelerating BERT Inference
  2. 2021Elbert: Fast Albert with Confidence-Window Based Early Exit
  3. 2025NeoBERT: A Next-Generation BERT
  4. 2024DE3-BERT: Distance-Enhanced Early Exiting for BERT based on Prototypical Networks
  5. 2024CEEBERT: Cross-Domain Inference in Early Exit BERT
  6. 2022HuBERT-EE: Early Exiting HuBERT for Efficient Speech Recognition
  7. 2022MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation
  8. 2022Token Dropping for Efficient BERT Pretraining