mlsanityData to 7 Oct 2026

Optimized Speculative Sampling for GPU Hardware Accelerators

arXiv 2406.11016 ↗First posted 16 Jun 2024cs.LG cs.CLcited by 2

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

In this work, we optimize speculative sampling for parallel hardware accelerators to improve sampling speed. We notice that substantial portions of the intermediate matrices necessary for speculative sampling can be computed concurrently. This allows us to distribute the workload across multiple GPU threads, enabling simultaneous operations on matrix segments within thread blocks. This results in profiling time improvements ranging from 6% to 13% relative to the baseline implementation, without compromising accuracy. To further accelerate speculative sampling, probability distributions parameterized by softmax are approximated by sigmoid. This approximation approach results in significantly greater relative improvements in profiling time, ranging from 37% to 94%, with a minor decline in accuracy. We conduct extensive experiments on both automatic speech recognition and summarization tasks to validate the effectiveness of our optimization methods.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 47 of its 74 references that are in the map. First posted on arXiv 16 Jun 2024.

Builds on

47 of 74 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  2. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  3. 2024Multi-Candidate Speculative Decoding12 Jan
  4. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  5. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  6. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  7. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  8. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  9. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation30 Mar

Other references · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  11. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  12. 2020GLU Variants Improve Transformercited by 1,724
26 more
  1. 2019Root Mean Square Layer Normalizationcited by 1,425
  2. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  3. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  4. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  5. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  6. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  7. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  8. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  9. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  10. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  11. 2021A White Paper on Neural Network Quantizationcited by 453
  12. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  13. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  14. 2021Block Pruning For Faster Transformerscited by 176
  15. 2018Online normalizer calculation for softmaxcited by 154
  16. 2021Self-attention Does Not Need O(n2) Memorycited by 152
  17. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147
  18. 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
  19. 2020Infinite attention: NNGP and NTK for deep attention networkscited by 117
  20. 2023Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labellingcited by 87
  21. 2016Efficient softmax approximation for GPUscited by 67
  22. 2023A Study on ReLU and Softmax in Transformercited by 53
  23. 2023Replacing softmax with ReLU in Vision Transformerscited by 43
  24. 2018Dissecting the NVIDIA Volta GPU Architecture via Microbenchmarkingcited by 35
  25. 2022Robust Training of Neural Networks Using Scale Invariant Architecturescited by 29
  26. 2024GRILLBot In Practice: Lessons and Tradeoffs Deploying Large Language Models for Adaptable Conversational Task Assistantscited by 6

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
  2. 2025SafeRoute: Adaptive Model Selection for Efficient and Accurate Safety Guardrails in Large Language Modelscited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2026FlashSampling: Fast and Memory-Efficient Exact Sampling
  2. 2025SIMPLE: Disaggregating Sampling from GPU Inference into a Decision Plane for Faster Distributed LLM Serving
  3. 2025EasySpec: Layer-Parallel Speculative Decoding for Efficient Multi-GPU Utilization
  4. 2025SpecOffload: Unlocking Latent GPU Capacity for LLM Inference on Resource-Constrained Devices
  5. 2026Parallel Track Transformers: Enabling Fast GPU Inference with Reduced Synchronization
  6. 2025FAST: An Efficient Scheduler for All-to-All GPU Communication
  7. 2023The Synergy of Speculative Decoding and Batching in Serving Large Language Models
  8. 2026The Economics of AI Decoding Chips: Rebalancing Compute, Capacity, and Bandwidth for Efficient LLM Inference