mlsanityData to 14 Oct 2026

S3D: A Simple and Cost-Effective Self-Speculative Decoding Scheme for Low-Memory GPUs

arXiv 2405.20314 ↗First posted 30 May 2024cs.CLcited by 6

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding (SD) has attracted a significant amount of research attention due to the substantial speedup it can achieve for LLM inference. However, despite the high speedups they offer, speculative decoding methods often achieve optimal performance on high-end devices or with a substantial GPU memory overhead. Given limited memory and the necessity of quantization, a high-performing model on a high-end GPU can slow down by up to 7 times. To this end, we propose Skippy Simultaneous Speculative Decoding (or S3D), a cost-effective self-speculative SD method based on simultaneous multi-token decoding and mid-layer skipping. When compared against recent effective open-source SD systems, our method has achieved one of the top performance-memory ratios while requiring minimal architecture changes and training data. Leveraging our memory efficiency, we created a smaller yet more effective SD model based on Phi-3. It is 1.4 to 2 times faster than the quantized EAGLE model and operates in half-precision while using less VRAM.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 31 of its 45 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

31 of 45 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting29 Apr
  2. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding19 Feb
  3. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Models16 Feb
  4. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding7 Feb
  5. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  6. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  7. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  8. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  9. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding9 Oct
  10. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  11. 2023Accelerating Transformer Inference for Translation via Parallel Decoding17 May
  12. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
3 more
  1. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  2. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation30 Mar
  3. 2021Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding9 Jun

Other references · 16

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  5. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  6. 2015Teaching Machines to Read and Comprehendcited by 1,049
  7. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 596
  8. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  9. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  10. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  11. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
  12. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
4 more
  1. 2021Block Pruning For Faster Transformerscited by 177
  2. 2015Structured Pruning of Deep Convolutional Neural Networkscited by 149
  3. 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118
  4. 2020Accelerating Feedforward Computation via Parallel Nonlinear Equation Solvingcited by 52

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 5.

6 papers.

  1. 2025Accelerating Large Language Model Reasoning via Speculative Searchcited by 16
  2. 2025SPIN: Accelerating Large Language Model Inference with Heterogeneous Speculative Modelscited by 9
  3. 2025DEL: Context-Aware Dynamic Exit Layer for Efficient Self-Speculative Decodingcited by 8
  4. 2025GoalLadder: Incremental Goal Discovery with Vision-Language Modelscited by 3
  5. 2024S2D: Sorted Speculative Decoding For More Efficient Deployment of Nested Large Language Modelscited by 1
  6. 2025Cross-Attention Speculative Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025EasySpec: Layer-Parallel Speculative Decoding for Efficient Multi-GPU Utilization
  2. 2025SpecMemo: Speculative Decoding is in Your Pocket
  3. 2025SpecOffload: Unlocking Latent GPU Capacity for LLM Inference on Resource-Constrained Devices
  4. 2024Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inference
  5. 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference
  6. 2024AMUSD: Asynchronous Multi-Device Speculative Decoding for LLM Acceleration
  7. 2023The Synergy of Speculative Decoding and Batching in Serving Large Language Models
  8. 2025CARD: A Cache-Assisted Parallel Speculative Decoding Framework via Query-and-Correct Paradigm for Accelerating LLM Inference