mlsanityData to 14 Oct 2026

SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference

arXiv 2307.02628 ↗First posted 5 Jul 2023cs.CLcited by 89

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Autoregressive large language models (LLMs) have made remarkable progress in various natural language generation tasks. However, they incur high computation cost and latency resulting from the autoregressive token-by-token generation. To address this issue, several approaches have been proposed to reduce computational cost using early-exit strategies. These strategies enable faster text generation using reduced computation without applying the full computation graph to each token. While existing token-level early exit methods show promising results for online inference, they cannot be readily applied for batch inferencing and Key-Value caching. This is because they have to wait until the last token in a batch exits before they can stop computing. This severely limits the practical application of such techniques. In this paper, we propose a simple and effective token-level early exit method, SkipDecode, designed to work seamlessly with batch inferencing and KV caching. It overcomes prior constraints by setting up a singular exit point for every token in a batch at each sequence position. It also guarantees a monotonic decrease in exit points, thereby eliminating the need to recompute KV Caches for preceding tokens. Rather than terminating computation prematurely as in prior works, our approach bypasses lower to middle layers, devoting most of the computational resources to upper layers, allowing later tokens to benefit from the compute expenditure by earlier tokens. Our experimental results show that SkipDecode can obtain 2x to 5x inference speedups with negligible regression across a variety of tasks. This is achieved using OPT models of 1.3 billion and 6.7 billion parameters, all the while being directly compatible with batching and KV caching optimization techniques.

Key facts. Cited by 89 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 21 of its 26 references that are in the map. First posted on arXiv 5 Jul 2023.

Builds on

21 of 26 references.

References · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  8. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  9. 2015Teaching Machines to Read and Comprehendcited by 1,049
  10. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  11. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 332
  12. 2022Confident Adaptive Language Modelingcited by 310
9 more
  1. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 273
  2. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 253
  3. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
  4. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 226
  5. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 195
  6. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  7. 2020Compression of Deep Learning Models for Text: A Surveycited by 64
  8. 2020CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cascadecited by 47
  9. 2022A Simple Hash-Based Early Exiting Approach For Language Understanding and Generationcited by 32

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 34; 2025: 27; 2026: 20.

89 papers.

Sort:

By area:

  1. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  2. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  3. 2023Challenges and Applications of Large Language Modelscited by 254
  4. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  5. 2023Efficient Large Language Models: A Surveycited by 186
  6. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
  7. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  8. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 125
  9. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 117
  10. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 116
  11. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  12. 2024SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blockscited by 97
38 more
  1. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  2. 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 82
  3. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 78
  4. 2025CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsificationcited by 74
  5. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 72
  6. 2025MoLe-VLA: Dynamic Layer-skipping Vision Language Action Model via Mixture-of-Layers for Efficient Robot Manipulationcited by 70
  7. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 69
  8. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  9. 2024VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computationcited by 55
  10. 2024A Survey on Transformer Compressioncited by 46
  11. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Modelscited by 38
  12. 2025A Survey of Large Language Model Empowered Agents for Recommendation and Search: Towards Next-Generation Information Retrievalcited by 33
  13. 2025System-1.5 Reasoning: Traversal in Language and Latent Spaces with Dynamic Shortcutscited by 27
  14. 2024CE-CoLLM: Efficient and Adaptive Large Language Models Through Cloud-Edge Collaborationcited by 25
  15. 2024FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skippingcited by 23
  16. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budgetcited by 23
  17. 2024Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decodingcited by 23
  18. 2024Accelerating Inference in Large Language Models with a Unified Layer Skipping Strategycited by 17
  19. 2025Think-at-Hard: Dynamic Looped Transformers for Improved Reasoningcited by 16
  20. 2025The Energy Cost of Reasoning: Analyzing Energy Usage in LLMs with Test-time Computecited by 15
  21. 2024EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Modelscited by 13
  22. 2025AdaSkip: Adaptive Sublayer Skipping for Accelerating Long-Context LLM Inferencecited by 12
  23. 2024Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modulescited by 11
  24. 2025CLaSp: In-Context Layer Skip for Self-Speculative Decodingcited by 11
  25. 2025SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decouplingcited by 11
  26. 2024Efficient Stagewise Pretraining via Progressive Subnetworkscited by 10
  27. 2026From LLMs to LRMs: Rethinking Pruning for Reasoning-Centric Modelscited by 10
  28. 2023Large Multimodal Model Compression via Efficient Pruning and Distillation at AntGroupcited by 8
  29. 2025HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Servingcited by 8
  30. 2025OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inferencecited by 8
  31. 2024Hierarchical Skip Decoding for Efficient Autoregressive Text Generationcited by 7
  32. 2024Duo-LLM: A Framework for Studying Adaptive Computation in Large Language Modelscited by 7
  33. 2025AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelismcited by 7
  34. 2025DIVE into MoE: Diversity-Enhanced Reconstruction of Large Language Models from Dense into Mixture-of-Expertscited by 7
  35. 2024QCQA: Quality and Capacity-aware grouped Query Attentioncited by 6
  36. 2024A Survey of Transformer-based Language Models with Focus on Efficiencycited by 6
  37. 2026Depth-adaptive Inference of Looped Language Models via Continuous Depth Batchingcited by 6
  38. 2024BlockLLM: Multi-tenant Finer-grained Serving for Large Language Modelscited by 5

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Time
  2. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding
  3. 2024An Efficient Inference Framework for Early-exit Large Language Models
  4. 2024Think Big, Generate Quick: LLM-to-SLM for Fast Autoregressive Decoding
  5. 2025READER: Retrieval-Assisted Drafter for Efficient LLM Inference
  6. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference
  7. 2025QuickSilver -- Speeding up LLM Inference through Dynamic Token Halting, KV Skipping, Contextual Token Fusion, and Adaptive Matryoshka Quantization
  8. 2025FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Prediction