mlsanityData to 7 Oct 2026

Speculative Streaming: Fast LLM Inference without Auxiliary Models

arXiv 2402.11131 ↗First posted 16 Feb 2024cs.CL cs.AI cs.LGcited by 37

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding is a prominent technique to speed up the inference of a large target language model based on predictions of an auxiliary draft model. While effective, in application-specific settings, it often involves fine-tuning both draft and target models to achieve high acceptance rates. As the number of downstream tasks grows, these draft models add significant complexity to inference systems. We propose Speculative Streaming, a single-model speculative decoding method that fuses drafting into the target model by changing the fine-tuning objective from next token prediction to future n-gram prediction. Speculative Streaming speeds up decoding by 1.8 - 3.1X in a diverse set of tasks, such as Summarization, Structured Queries, and Meaning Representation, without sacrificing generation quality. Additionally, Speculative Streaming is parameter-efficient. It achieves on-par/higher speed-ups than Medusa-style architectures while using ~10000X fewer extra parameters, making it well-suited for resource-constrained devices.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 32 of its 40 references that are in the map. First posted on arXiv 16 Feb 2024.

Builds on

32 of 40 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding7 Feb
  2. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  3. 2023SpecTr: Fast Speculative Decoding via Optimal Transport23 Oct
  4. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  5. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  6. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  7. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  11. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  12. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
13 more
  1. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  2. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  3. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  4. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  5. 2022Efficiently Scaling Transformer Inferencecited by 529
  6. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  7. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  8. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  9. 2020ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingcited by 221
  10. 2021DialogSum: A Real-Life Scenario Dialogue Summarization Datasetcited by 181
  11. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  12. 2023Future Lens: Anticipating Subsequent Tokens from a Single Hidden Statecited by 116
  13. 2019Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challengecited by 86

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 17; 2026: 6.

37 papers.

Sort:

By area:

  1. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  2. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengthscited by 73
  3. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  4. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignmentcited by 59
  5. 2025Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potentialcited by 46
  6. 2024Recurrent Drafter for Fast Speculative Decoding in Large Language Modelscited by 35
  7. 2024Dynamic Speculation Lookahead Accelerates Speculative Decoding of Large Language Modelscited by 29
  8. 2025CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routingcited by 29
  9. 2024A Theoretical Perspective for Speculative Decoding Algorithmcited by 28
  10. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cachecited by 26
  11. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
  12. 2024ParallelSpec: Parallel Drafter for Efficient Speculative Decodingcited by 20
25 more
  1. 2024Remote Timing Attacks on Efficient Language Model Inferencecited by 13
  2. 2024The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculationcited by 13
  3. 2024Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafterscited by 12
  4. 2024Clover: Regressive Lightweight Speculative Decoding with Sequential Knowledgecited by 10
  5. 2025KNN-SSD: Enabling Dynamic Self-Speculative Decoding via Nearest Neighbor Layer Set Optimizationcited by 10
  6. 2025SpecEdge: Scalable Edge-Assisted Serving Framework for Interactive LLMscited by 10
  7. 2024Extracting Paragraphs from LLM Token Activationscited by 9
  8. 2025ML-SpecQD: Multi-Level Speculative Decoding with Quantized Draftscited by 9
  9. 2025Guided Speculative Inference for Efficient Test-Time Alignment of LLMscited by 9
  10. 2024Faster Language Models with Better Multi-Token Prediction Using Tensor Decompositioncited by 7
  11. 2024S3D: A Simple and Cost-Effective Self-Speculative Decoding Scheme for Low-Memory GPUscited by 6
  12. 2025Collaborative Speculative Inference for Efficient LLM Inference Servingcited by 5
  13. 2025Token-Driven GammaTune: Adaptive Calibration for Enhanced Speculative Decodingcited by 5
  14. 2025OmniDraft: A Cross-vocabulary, Online Adaptive Drafter for On-device Speculative Decodingcited by 4
  15. 2025Small Language Models (SLMs) Can Still Pack a Punch: A survey (updated 2026)cited by 2
  16. 2025Semi-Clairvoyant Scheduling of Speculative Decoding Requests to Minimize LLM Inference Latencycited by 2
  17. 2025ParaScopes: What do Language Models Activations Encode About Future Text?cited by 2
  18. 2025M2R2: Mixture of Multi-Rate Residuals for Efficient Transformer Inferencecited by 1
  19. 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inferencecited by 1
  20. 2026Efficient Training-Free Multi-Token Prediction via Embedding-Space Probingcited by 0
  21. 2026Exploring and Developing a Pre-Model Safeguard with Draft Modelscited by 0
  22. 2026CONCORD: Asynchronous Sparse Aggregation for Device-Cloud RAG under Document Isolationcited by 0
  23. 2026Leaky Language Models: Stealing Architecture and Inference Optimizations via Per-Token Timingcited by 0
  24. 2026Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decodingcited by 0
  25. 2026SEED: Self-Speculative Decoding via Implicit Encoder-Decodercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Speculative Decoding: Performance or Illusion?
  2. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding
  3. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  4. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
  5. 2024Decoding Speculative Decoding
  6. 2024Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement
  7. 2026Faster LLM Inference via Sequential Monte Carlo
  8. 2024SSSD: Simply-Scalable Speculative Decoding