mlsanityData to 14 Oct 2026

Magic Pyramid: Accelerating Inference with Early Exiting and Token Pruning

arXiv 2111.00230 ↗First posted 30 Oct 2021cs.CLcited by 16

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

Pre-training and then fine-tuning large language models is commonly used to achieve state-of-the-art performance in natural language processing (NLP) tasks. However, most pre-trained models suffer from low inference speed. Deploying such large models to applications with latency constraints is challenging. In this work, we focus on accelerating the inference via conditional computations. To achieve this, we propose a novel idea, Magic Pyramid (MP), to reduce both width-wise and depth-wise computation via token pruning and early exiting for Transformer-based models, particularly BERT. The former manages to save the computation via removing non-salient tokens, while the latter can fulfill the computation reduction by terminating the inference early before reaching the final layer, if the exiting condition is met. Our empirical studies demonstrate that compared to previous state of arts, MP is not only able to achieve a speed-adjustable inference but also to surpass token pruning and early exiting by reducing up to 70% giga floating point operations (GFLOPs) with less than 0.5% accuracy drop. Token pruning and early exiting express distinctive preferences to sequences with different lengths. However, MP is capable of achieving an average of 8.06x speedup on two popular text classification tasks, regardless of the sizes of the inputs.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 20 of its 27 references that are in the map. First posted on arXiv 30 Oct 2021.

Builds on

20 of 27 references.

References · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  8. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  9. 2019Are Sixteen Heads Really Better than One?cited by 816
  10. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 596
  11. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  12. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
8 more
  1. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  2. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  3. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  4. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
  5. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  6. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 226
  7. 2021Learned Token Pruning for Transformerscited by 139
  8. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 126

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 2; 2024: 7; 2025: 2; 2026: 2.

16 papers.

Sort:

By area:

  1. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  2. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  3. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  4. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  5. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 66
  6. 2022Transkimmer: Transformer Learns to Layer-wise Skimcited by 24
  7. 2022Efficiently Controlling Multiple Risks with Pareto Testingcited by 22
  8. 2023IMBERT: Making BERT Immune to Insertion-based Backdoor Attackscited by 15
  9. 2024CEEBERT: Cross-Domain Inference in Early Exit BERTcited by 12
  10. 2024Conditional computation in neural networks: principles and research trendscited by 10
  11. 2022Task-specific Compression for Multi-task Language Models using Attribution-based Pruningcited by 7
  12. 2024DAdEE: Unsupervised Domain Adaptation in Early Exit PLMscited by 5
4 more
  1. 2024RAEE: A Robust Retrieval-Augmented Early Exit Framework for Efficient Inferencecited by 4
  2. 2025Beyond Greedy Exits: Improved Early Exit Decisions for Risk Control and Reliabilitycited by 3
  3. 2026General learned delegation by clonescited by 0
  4. 2026Adaptive Computation with Conformal Certificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026POP: Prefill-Only Pruning for Efficient Large Model Inference
  2. 2023Constraint-aware and Ranking-distilled Token Pruning for Efficient Transformer Inference
  3. 2024Dynamic Vocabulary Pruning in Early-Exit LLMs
  4. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models
  5. 2024Accelerating Large Language Model Inference with Self-Supervised Early Exits
  6. 2025QuickSilver -- Speeding up LLM Inference through Dynamic Token Halting, KV Skipping, Contextual Token Fusion, and Adaptive Matryoshka Quantization
  7. 2025Leveraging KV Similarity for Online Structured Pruning in LLMs
  8. 2022Fast DistilBERT on CPUs