mlsanityData to 7 Oct 2026

Token Dropping for Efficient BERT Pretraining

arXiv 2203.13240 ↗First posted 24 Mar 2022cs.CL cs.LGcited by 42

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

Transformer-based models generally allocate the same amount of computation for each token in a given sequence. We develop a simple but effective "token dropping" method to accelerate the pretraining of transformer models, such as BERT, without degrading its performance on downstream tasks. In short, we drop unimportant tokens starting from an intermediate layer in the model to make the model focus on important tokens; the dropped tokens are later picked up by the last layer of the model so that the model still produces full-length sequences. We leverage the already built-in masked language modeling (MLM) loss to identify unimportant tokens with practically no computational overhead. In our experiments, this simple approach reduces the pretraining cost of BERT by 25% while achieving similar overall fine-tuning performance on standard downstream tasks.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 36 of its 51 references that are in the map. First posted on arXiv 24 Mar 2022.

Builds on

36 of 51 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  11. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  12. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
24 more
  1. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  2. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  3. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  4. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  5. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  6. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  7. 2018Universal Transformerscited by 613
  8. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  9. 2017Non-Autoregressive Neural Machine Translationcited by 457
  10. 2021Dynamic Neural Networks: A Surveycited by 405
  11. 2020AraBERT: Transformer-based Model for Arabic Language Understandingcited by 359
  12. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  13. 2019Depth-Adaptive Transformercited by 227
  14. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  15. 2019The Evolved Transformercited by 180
  16. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
  17. 2019Are All Layers Created Equal?cited by 94
  18. 2020Accelerating Training of Transformer-Based Language Models with Progressive Layer Droppingcited by 93
  19. 2020ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translationcited by 34
  20. 2021Pre-trained Language Model based Ranking in Baidu Searchcited by 33
  21. 2018A Stable and Effective Learning Strategy for Trainable Greedy Decodingcited by 12
  22. 2019Improving Joint Training of Inference Networks and Structured Prediction Energy Networkscited by 6
  23. 2020Taking Notes on the Fly Helps BERT Pre-trainingcited by 5
  24. 2021Amortized Noisy Channel Neural Machine Translationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 7; 2024: 14; 2025: 13; 2026: 2.

42 papers.

Sort:

By area:

  1. 2025TokenSkip: Controllable Chain-of-Thought Compression in LLMscited by 246
  2. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  3. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 86
  4. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  5. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 81
  6. 2024Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMscited by 50
  7. 2024Efficient Large Multi-modal Models via Visual Context Compressioncited by 46
  8. 2024Checkpoint Merging via Bayesian Optimization in LLM Pretrainingcited by 27
  9. 2024Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decodingcited by 21
  10. 2025SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMscited by 17
  11. 2022Multi-CLS BERT: An Efficient Alternative to Traditional Ensemblingcited by 14
  12. 2022DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routingcited by 13
30 more
  1. 2022Random-LTD: Random and Layerwise Token Dropping Brings Efficient Training for Large-scale Transformerscited by 11
  2. 2023Revisiting Token Dropping Strategy in Efficient BERT Pretrainingcited by 11
  3. 2025Not All Errors Are Created Equal: ASCoT Addresses Late-Stage Fragility in Efficient LLM Reasoningcited by 11
  4. 2024Conditional computation in neural networks: principles and research trendscited by 10
  5. 2023GQKVA: Efficient Pre-training of Transformers by Grouping Queries, Keys, and Valuescited by 9
  6. 2025Not All Tokens Matter: Towards Efficient LLM Reasoning via Token Significance in Reinforcement Learningcited by 9
  7. 2022hmBERT: Historical Multilingual Language Models for Named Entity Recognitioncited by 8
  8. 2022Mask More and Mask Later: Efficient Pre-training of Masked Language Models by Disentangling the [MASK] Tokencited by 7
  9. 2025Accelerating Adaptive Retrieval Augmented Generation via Instruction-Driven Representation Reduction of Retrieval Overlapscited by 6
  10. 2024Train-Attention: Meta-Learning Where to Focus in Continual Knowledge Learningcited by 5
  11. 2025Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Predictioncited by 5
  12. 2023Parameter-Efficient Fine-Tuning with Layer Pruning on Free-Text Sequence-to-Sequence Modelingcited by 3
  13. 2024Efficient Backpropagation with Variance-Controlled Adaptive Samplingcited by 3
  14. 2024Synthetic4Health: Generating Annotated Synthetic Clinical Letterscited by 3
  15. 2025Aggregation Buffer: Revisiting DropEdge with a New Parameter Blockcited by 3
  16. 2023Weighted Sampling for Masked Language Modelingcited by 2
  17. 2024Ignore Me But Don't Replace Me: Utilizing Non-Linguistic Elements for Pretraining on the Cybersecurity Domaincited by 2
  18. 2025ResidualViT for Efficient Temporally Dense Video Encodingcited by 2
  19. 2025Efficient Vision-Language Reasoning via Adaptive Token Pruningcited by 2
  20. 2023Input-length-shortening and text generation via attention valuescited by 1
  21. 2023Improve Supervised Representation Learning with Masked Image Modelingcited by 1
  22. 2025Unlocking Full Efficiency of Token Filtering in Large Language Model Trainingcited by 1
  23. 2025Rethinking Sparse Mixture of Experts from a Unified Perspectivecited by 1
  24. 2025ESLM: Risk-Averse Selective Language Modeling for Efficient Pretrainingcited by 1
  25. 2024A Multi-Level Framework for Accelerating Training Transformer Modelscited by 0
  26. 2024Switchable Decision: Dynamic Neural Generation Networkscited by 0
  27. 2024Federating to Grow Transformers with Constrained Resources without Model Sharingcited by 0
  28. 2025Fast Vision Mamba: Pooling Spatial Dimensions for Accelerated Processingcited by 0
  29. 2026TokenSeek: Memory Efficient Fine Tuning via Instance-Aware Token Ditchingcited by 0
  30. 2026YoNER: A New Yor`ub'a Multi-domain Named Entity Recognition Datasetcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Constraint-aware and Ranking-distilled Token Pruning for Efficient Transformer Inference
  2. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models
  3. 2022Pretraining Without Attention
  4. 2021Accelerating BERT Inference for Sequence Labeling via Early-Exit
  5. 2021TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference
  6. 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformations
  7. 2021Maximizing Efficiency of Language Model Pre-training for Learning Representation
  8. 2025Lossless Token Sequence Compression via Meta-Tokens