mlsanityData to 7 Oct 2026

Speculative Decoding with Big Little Decoder

arXiv 2302.07863 ↗First posted 15 Feb 2023cs.CLcited by 163

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

The recent emergence of Large Language Models based on the Transformer architecture has enabled dramatic advancements in the field of Natural Language Processing. However, these models have long inference latency, which limits their deployment and makes them prohibitively expensive for various real-time applications. The inference latency is further exacerbated by autoregressive generative tasks, as models need to run iteratively to generate tokens sequentially without leveraging token-level parallelization. To address this, we propose Big Little Decoder (BiLD), a framework that can improve inference efficiency and latency for a wide range of text generation applications. The BiLD framework contains two models with different sizes that collaboratively generate text. The small model runs autoregressively to generate text with a low inference cost, and the large model is only invoked occasionally to refine the small model's inaccurate predictions in a non-autoregressive manner. To coordinate the small and large models, BiLD introduces two simple yet effective policies: (1) the fallback policy that determines when to hand control over to the large model; and (2) the rollback policy that determines when the large model needs to correct the small model's inaccurate predictions. To evaluate our framework across different tasks and models, we apply BiLD to various text generation scenarios encompassing machine translation on IWSLT 2017 De-En and WMT 2014 De-En, and summarization on XSUM and CNN/DailyMail. On an NVIDIA T4 GPU, our framework achieves a speedup of up to 2.12x speedup with minimal generation quality degradation. Furthermore, our framework is fully plug-and-play and can be applied without any modifications in the training process or model architecture. Our code is open-sourced

Key facts. Cited by 163 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 74 of its 85 references that are in the map. First posted on arXiv 15 Feb 2023.

Builds on

74 of 85 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  2. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 72

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2017On Calibration of Modern Neural Networkscited by 4,860
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2016Pointer Sentinel Mixture Modelscited by 2,992
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
38 more
  1. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  2. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  3. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  6. 2020Reformer: The Efficient Transformercited by 1,658
  7. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  8. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  9. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  10. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  11. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  12. 2015Teaching Machines to Read and Comprehendcited by 1,039
  13. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  14. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  15. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  16. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  17. 2019Are Sixteen Heads Really Better than One?cited by 803
  18. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  19. 2022Efficiently Scaling Transformer Inferencecited by 529
  20. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  21. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  22. 2019The State of Sparsity in Deep Neural Networkscited by 471
  23. 2017Non-Autoregressive Neural Machine Translationcited by 457
  24. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  25. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  26. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  27. 2022Confident Adaptive Language Modelingcited by 297
  28. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  29. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  30. 2019Depth-Adaptive Transformercited by 227
  31. 2019Levenshtein Transformercited by 197
  32. 2018Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinementcited by 194
  33. 2019The Evolved Transformercited by 180
  34. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  35. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  36. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 152
  37. 2020HAT: Hardware-Aware Transformers for Efficient Natural Language Processingcited by 136
  38. 2020Lite Transformer with Long-Short Range Attentioncited by 127

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 58; 2025: 64; 2026: 35.

163 papers.

Sort:

By area:

  1. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  2. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  3. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  4. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  5. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 266
  6. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  7. 2023Challenges and Applications of Large Language Modelscited by 251
  8. 2023Efficient Large Language Models: A Surveycited by 183
  9. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  10. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  11. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  12. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
38 more
  1. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 98
  2. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
  3. 2024GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decodingcited by 74
  4. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengthscited by 73
  5. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  6. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 69
  7. 2025BEST-Route: Adaptive LLM Routing with Test-Time Optimal Computecited by 63
  8. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 60
  9. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignmentcited by 59
  10. 2024Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMscited by 50
  11. 2024The Larger the Better? Improved LLM Code-Generation via Budget Reallocationcited by 41
  12. 2023Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decodingcited by 40
  13. 2024Faster Cascades via Speculative Decodingcited by 38
  14. 2025ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verificationcited by 35
  15. 2024Routoo: Learning to Route to Large Language Models Effectivelycited by 34
  16. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 30
  17. 2024A Theoretical Perspective for Speculative Decoding Algorithmcited by 28
  18. 2024Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacementcited by 27
  19. 2024Fast yet Safe: Early-Exiting with Risk Controlcited by 27
  20. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cachecited by 26
  21. 2024AdaEDL: Early Draft Stopping for Speculative Decoding of Large Language Models via an Entropy-based Lower Bound on Token Acceptance Probabilitycited by 24
  22. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decodingcited by 22
  23. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
  24. 2024Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decodingcited by 21
  25. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 21
  26. 2024Falcon: Faster and Parallel Inference of Large Language Models through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Treecited by 20
  27. 2025SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMscited by 20
  28. 2025LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verificationcited by 19
  29. 2025SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruningcited by 19
  30. 2024Continuous Speculative Decoding for Autoregressive Image Generationcited by 18
  31. 2025AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Servingcited by 18
  32. 2024Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inferencecited by 16
  33. 2024ESPACE: Dimensionality Reduction of Activations for Model Compressioncited by 15
  34. 2025GRIFFIN: Effective Token Alignment for Faster Speculative Decodingcited by 15
  35. 2025Ultra-Fast Language Generation via Discrete Diffusion Divergence Instructcited by 15
  36. 2026SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decodingcited by 15
  37. 2025TETRIS: Optimal Draft Token Selection for Batch Speculative Decodingcited by 14
  38. 2025Fuzzy Speculative Decoding for a Tunable Accuracy-Runtime Tradeoffcited by 14

50 most cited of 163 All 163 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Minions: Accelerating Large Language Model Inference with Aggregated Speculative Execution
  2. 2025Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies
  3. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  4. 2025READER: Retrieval-Assisted Drafter for Efficient LLM Inference
  5. 2024S2D: Sorted Speculative Decoding For More Efficient Deployment of Nested Large Language Models
  6. 2026LongSpark: Efficient speculative decoding with a fixed-cost parallel drafter
  7. 2024SSSD: Simply-Scalable Speculative Decoding
  8. 2024Decoding Speculative Decoding