mlsanityData to 7 Oct 2026

SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices

arXiv 2406.02532 ↗First posted 4 Jun 2024cs.CLcited by 45

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

As large language models gain widespread adoption, running them efficiently becomes crucial. Recent works on LLM inference use speculative decoding to achieve extreme speedups. However, most of these works implicitly design their algorithms for high-end datacenter hardware. In this work, we ask the opposite question: how fast can we run LLMs on consumer machines? Consumer GPUs can no longer fit the largest available models (50B+ parameters) and must offload them to RAM or SSD. When running with offloaded parameters, the inference engine can process batches of hundreds or thousands of tokens at the same time as just one token, making it a natural fit for speculative decoding. We propose SpecExec (Speculative Execution), a simple parallel decoding method that can generate up to 20 tokens per target model iteration for popular LLM families. It utilizes the high spikiness of the token probabilities distribution in modern LLMs and a high degree of alignment between model output probabilities. SpecExec takes the most probable tokens continuation from the draft model to build a "cache" tree for the target model, which then gets validated in a single pass. Using SpecExec, we demonstrate inference of 50B+ parameter LLMs on consumer GPUs with RAM offloading at 4-6 tokens per second with 4-bit quantization or 2-3 tokens per second with 16-bit weights.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 39 of its 68 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

39 of 68 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Beyond the Speculative Game: A Survey of Speculative Execution in Large Language Models23 Apr
  2. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  3. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  4. 2023REST: Retrieval-Based Speculative Decoding14 Nov
  5. 2023SpecTr: Fast Speculative Decoding via Optimal Transport23 Oct
  6. 2023Online Speculative Decoding11 Oct
  7. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  8. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  9. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 30

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2016Pointer Sentinel Mixture Modelscited by 2,992
  6. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  10. 2024Mixtral of Expertscited by 1,785
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  12. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
18 more
  1. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  2. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  3. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  4. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  5. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  6. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  7. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  8. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  9. 2022A Close Look into the Calibration of Pre-trained Language Modelscited by 57
  10. 2023LLMCad: Fast and Scalable On-device Large Language Model Inferencecited by 45
  11. 2020Training Large Neural Networks with Constant Memory using a New Execution Algorithmcited by 34
  12. 2021Prevent the Language Model from being Overconfident in Neural Machine Translationcited by 34
  13. 2015Gunrock: A High-Performance Graph Processing Library on the GPUcited by 14
  14. 2015The GAP Benchmark Suitecited by 14
  15. 2020To Push or To Pull: On Reducing Communication and Synchronization in Graph Computationscited by 11
  16. 2019Optimizing Ordered Graph Algorithms with GraphItcited by 2
  17. 2016Parallel Shortest-Paths Using Radius Steppingcited by 1
  18. 2019A parallel priority queue with fast updates for GPU architecturescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 27; 2026: 14.

45 papers.

Sort:

By area:

  1. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  2. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 318
  3. 2024Speculative Diffusion Decoding: Accelerating Language Generation through Diffusioncited by 57
  4. 2025DiffuSpec: Unlocking Diffusion Language Models for Speculative Decodingcited by 40
  5. 2025APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generationcited by 27
  6. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
  7. 2025MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoEcited by 17
  8. 2025GRIFFIN: Effective Token Alignment for Faster Speculative Decodingcited by 15
  9. 2024Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inferencecited by 13
  10. 2025AutoJudge: Judge Decoding Without Manual Annotationcited by 13
  11. 2026Speculative Speculative Decodingcited by 13
  12. 2025SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inferencecited by 12
33 more
  1. 2025SpecEdge: Scalable Edge-Assisted Serving Framework for Interactive LLMscited by 10
  2. 2025SPIN: Accelerating Large Language Model Inference with Heterogeneous Speculative Modelscited by 9
  3. 2025CLaSp: In-Context Layer Skip for Self-Speculative Decodingcited by 9
  4. 2025Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decodingcited by 9
  5. 2025AdaSPEC: Selective Knowledge Distillation for Efficient Speculative Decoderscited by 9
  6. 2025C2T: A Classifier-Based Tree Construction Method in Speculative Decodingcited by 8
  7. 2025STree: Speculative Tree Decoding for Hybrid State-Space Modelscited by 8
  8. 2025LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Serverscited by 8
  9. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decodingcited by 7
  10. 2025ActionFlow: A Pipelined Action Acceleration for Vision Language Models on Edgecited by 6
  11. 2025A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthinesscited by 5
  12. 2026HSD: Training-Free Acceleration for Document Parsing Vision-Language Models with Hierarchical Speculative Decodingcited by 5
  13. 2026MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarioscited by 5
  14. 2025TokenSwift: Lossless Acceleration of Ultra Long Sequence Generationcited by 4
  15. 2025Attention Misses Visual Risk: Risk-Adaptive Steering for Multimodal Safety Alignmentcited by 4
  16. 2025EasySpec: Layer-Parallel Speculative Decoding for Efficient Multi-GPU Utilizationcited by 3
  17. 2025Sparse-to-Dense: A Free Lunch for Lossless Acceleration of Video Understanding in LLMscited by 3
  18. 2026SpecMoE: A Fast and Efficient Mixture-of-Experts Inference via Self-Assisted Speculative Decodingcited by 3
  19. 2025HeteroSpec: Leveraging Contextual Heterogeneity for Efficient Speculative Decodingcited by 2
  20. 2025Semi-Clairvoyant Scheduling of Speculative Decoding Requests to Minimize LLM Inference Latencycited by 2
  21. 2026PRESTO: Prefix-Aligned Tree Drafting for Diffusion Speculative Decodingcited by 2
  22. 2025P/D-Device: Disaggregated Large Language Model between Cloud and Devicescited by 1
  23. 2025Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decodingcited by 1
  24. 2025Polybasic Speculative Decoding Through a Theoretical Perspectivecited by 1
  25. 2026Quasar: Quantized Self-Speculative Acceleration for Rapid Inference via Memory-Efficient Verificationcited by 1
  26. 2026FASER: Fine-Grained Phase Management for Speculative Decoding in Dynamic LLM Servingcited by 1
  27. 2026WANSpec: Leveraging Global Compute Capacity for LLM Inferencecited by 0
  28. 2026Position-Aware Drafting for Inference Acceleration in LLM-Based Generative List-Wise Recommendationcited by 0
  29. 2026Performance-Driven Policy Optimization for Speculative Decoding with Adaptive Windowingcited by 0
  30. 2026AdaptiveSD A Stability-Aware, Runtime-Adaptive Speculative Decoding Framework with Multi-Policy Orchestration for CPU-Constrained LLM Inferencecited by 0
  31. 2026BALANCE: Hybrid Autoregressive-Speculative LLM Inference at the Network Edgecited by 0
  32. 2026MemSpec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge Devicescited by 0
  33. 2026TreeGraft: Adaptive Multi-Drafter Grafting for Tree-Based Speculative Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference
  2. 2025Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios
  3. 2025CARD: A Cache-Assisted Parallel Speculative Decoding Framework via Query-and-Correct Paradigm for Accelerating LLM Inference
  4. 2026SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inference
  5. 2023The Synergy of Speculative Decoding and Batching in Serving Large Language Models
  6. 2025SpecMemo: Speculative Decoding is in Your Pocket
  7. 2026CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration
  8. 2026Compiler-Assisted Speculative Sampling for Accelerated LLM Inference on Heterogeneous Edge Devices