mlsanityData to 14 Oct 2026

TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference

arXiv 2105.11618 ↗First posted 25 May 2021cs.CLcited by 40

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

Existing pre-trained language models (PLMs) are often computationally expensive in inference, making them impractical in various resource-limited real-world applications. To address this issue, we propose a dynamic token reduction approach to accelerate PLMs' inference, named TR-BERT, which could flexibly adapt the layer number of each token in inference to avoid redundant calculation. Specially, TR-BERT formulates the token reduction process as a multi-step token selection problem and automatically learns the selection strategy via reinforcement learning. The experimental results on several downstream NLP tasks show that TR-BERT is able to speed up BERT by 2-5 times to satisfy various performance demands. Moreover, TR-BERT can also achieve better performance with less computation in a suite of long-text tasks since its token-level layer number adaption greatly accelerates the self-attention operation in PLMs. The source code and experiment details of this paper can be obtained from https://github.com/thunlp/TR-BERT.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 43 of its 59 references that are in the map. First posted on arXiv 25 May 2021.

Builds on

43 of 59 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  9. 2020Longformer: The Long-Document Transformercited by 3,413
  10. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  12. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
31 more
  1. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  2. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  3. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  4. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  5. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  6. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  7. 2019Are Sixteen Heads Really Better than One?cited by 816
  8. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  9. 2018Universal Transformerscited by 647
  10. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  11. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  12. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  13. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  14. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 332
  15. 2016NewsQA: A Machine Comprehension Datasetcited by 331
  16. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  17. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
  18. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 226
  19. 2017Constructing Datasets for Multi-hop Reading Comprehension Across Documentscited by 190
  20. 2019MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehensioncited by 184
  21. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 162
  22. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 148
  23. 2019Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentencecited by 122
  24. 2017Simple and Effective Multi-Paragraph Reading Comprehensioncited by 120
  25. 2019Open Sesame: Getting Inside BERT's Linguistic Knowledgecited by 120
  26. 2019Semantics-aware BERT for Language Understandingcited by 112
  27. 2020Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Searchcited by 76
  28. 2019MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehensioncited by 60
  29. 2019Revealing the Importance of Semantic Retrieval for Machine Reading at Scalecited by 50
  30. 2020DeFormer: Decomposing Pre-trained Transformers for Faster Question Answeringcited by 29
  31. 2020PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Eliminationcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 9; 2023: 12; 2024: 8; 2025: 4; 2026: 4.

40 papers.

Sort:

By area:

  1. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  2. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 152
  3. 2021Learned Token Pruning for Transformerscited by 139
  4. 2024Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Modelscited by 126
  5. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  6. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  7. 2023Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformerscited by 62
  8. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 50
  9. 2021Packed Levitated Marker for Entity and Relation Extractioncited by 49
  10. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  11. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  12. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
28 more
  1. 2022A Survey on Dynamic Neural Networks for Natural Language Processingcited by 24
  2. 2022Transkimmer: Transformer Learns to Layer-wise Skimcited by 24
  3. 2022Efficiently Controlling Multiple Risks with Pareto Testingcited by 22
  4. 2022AdapLeR: Speeding up Inference by Adaptive Length Reductioncited by 19
  5. 2024Bridging Local Details and Global Context in Text-Attributed Graphscited by 15
  6. 2022Pyramid-BERT: Reducing Complexity via Successive Core-set based Token Selectioncited by 13
  7. 2023ConsistentEE: A Consistent and Hardness-Guided Early Exiting Method for Accelerating Language Models Inferencecited by 13
  8. 2022Sparse Attention Acceleration with Synergistic In-Memory Pruning and On-Chip Recomputationcited by 12
  9. 2025Rethinking Chain-of-Thought Reasoning for Videoscited by 9
  10. 2022Accelerating Attention through Gradient-Based Learned Runtime Pruningcited by 8
  11. 2024Language Model Guided Interpretable Video Action Reasoningcited by 8
  12. 2023SmartTrim: Adaptive Tokens and Attention Pruning for Efficient Vision-Language Modelscited by 7
  13. 2023Constraint-aware and Ranking-distilled Token Pruning for Efficient Transformer Inferencecited by 6
  14. 2024CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modelingcited by 4
  15. 2023AttMEMO : Accelerating Transformers with Memoization on Big Memory Systemscited by 3
  16. 2025SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Modelingcited by 3
  17. 2026SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Modelscited by 3
  18. 2023Infor-Coef: Information Bottleneck-based Dynamic Token Downsampling for Compact and Efficient language modelcited by 2
  19. 2023You Only Forward Once: Prediction and Rationalization in A Single Forward Passcited by 2
  20. 2026Control Reinforcement Learning: Interpretable Token-Level Steering of LLMs via Sparse Autoencoder Featurescited by 2
  21. 2023Variator: Accelerating Pre-trained Models with Plug-and-Play Compression Modulescited by 1
  22. 2023No-Skim: Towards Efficiency Robustness Evaluation on Skimming-based Language Modelscited by 1
  23. 2024Practical token pruning for foundation models in few-shot conversational virtual assistant systemscited by 1
  24. 2024Self-Satisfied: An end-to-end framework for SAT generation and predictioncited by 1
  25. 2022An Efficient End-to-End Transformer with Progressive Tri-modal Attention for Multi-modal Emotion Recognitioncited by 0
  26. 2023Harnessing Attention Mechanisms: Efficient Sequence Reduction using Attention-based Autoencoderscited by 0
  27. 2026Sparse Token Routing in Efficient Transformerscited by 0
  28. 2026Dependency-Aware Chain-of-Thought Compression for Financial Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Token Dropping for Efficient BERT Pretraining
  2. 2023Revisiting Token Dropping Strategy in Efficient BERT Pretraining
  3. 2023SmartBERT: A Promotion of Dynamic Early Exiting Mechanism for Accelerating BERT Inference
  4. 2021Accelerating BERT Inference for Sequence Labeling via Early-Exit
  5. 2021Elbert: Fast Albert with Confidence-Window Based Early Exit
  6. 2021RefBERT: Compressing BERT by Referencing to Pre-computed Representations
  7. 2024CEEBERT: Cross-Domain Inference in Early Exit BERT
  8. 2022MKQ-BERT: Quantized BERT with 4-bits Weights and Activations