mlsanityData to 7 Oct 2026

Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference

arXiv 2510.13161 ↗First posted 15 Oct 2025cs.CLcited by 1

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding accelerates LLM inference by using a draft model to look ahead, but gains are capped by the cost of autoregressive draft generation: increasing draft size elevates acceptance rates but introduces additional latency overhead exacerbating the speed-accuracy tradeoff. Prior methods (Medusa, Hydra, EAGLE) partially reduce draft cost but either degrade acceptance or introduce overheads that limit scaling. We present Mirror Speculative Decoding (Mirror-SD), an inference algorithm that breaks the latency-acceptance tradeoff. Mirror-SD launches branch-complete rollouts from early-exit signals in parallel with the target model's suffix and explicitly maps computation across heterogeneous accelerators (GPU and NPU) to exploit cross-device parallelism. The draft speculates forward continuations for the target to verify, while the target simultaneously speculates correction paths for the draft, converting speculation into two complementary execution pipelines. To further cut draft latency without weakening acceptance semantics, we add speculative streaming so the draft emits multiple tokens per step. This dual strategy of parallel heterogeneous execution plus multi-token speculative streaming pushes speculative decoding toward its ideal regime of high acceptance with low overhead. On SpecBench with server-scale models from 14B to 66B parameters, Mirror-SD delivers consistent end-to-end gains, achieving 2.8x-5.8x wall-time speedups across diverse tasks and a 30% average relative improvement over the strongest baseline, EAGLE3.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 67 of its 91 references that are in the map. First posted on arXiv 15 Oct 2025.

Builds on

67 of 91 references.

Closest ancestors · 23

Same direction, newest first

  1. 2025Training Domain Draft Models for Speculative Decoding: Best Practices and Insights10 Mar
  2. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
  3. 2025M2R2: Mixture of Multi-Rate Residuals for Efficient Transformer Inference4 Feb
  4. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  5. 2024Direct Alignment of Draft Model for Speculative Decoding with Chat-Fine-Tuned LLMs29 Feb
  6. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding19 Feb
  7. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Models16 Feb
  8. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding7 Feb
  9. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  10. 2024Decoding Speculative Decoding2 Feb
  11. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  12. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
11 more
  1. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  2. 2023PaSS: Parallel Speculative Sampling22 Nov
  3. 2023REST: Retrieval-Based Speculative Decoding14 Nov
  4. 2023SpecTr: Fast Speculative Decoding via Optimal Transport23 Oct
  5. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  6. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  7. 2023SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference5 Jul
  8. 2023Accelerating Transformer Inference for Translation via Parallel Decoding17 May
  9. 2023Speculative Decoding with Big Little Decoder15 Feb
  10. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  11. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
32 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  3. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  4. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  5. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  6. 2023BloombergGPT: A Large Language Model for Financecited by 923
  7. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  8. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  9. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  10. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  11. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  12. 2022Efficiently Scaling Transformer Inferencecited by 529
  13. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  14. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  15. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  16. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  17. 2022Confident Adaptive Language Modelingcited by 297
  18. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 247
  19. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  20. 2020ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingcited by 221
  21. 2021DialogSum: A Real-Life Scenario Dialogue Summarization Datasetcited by 181
  22. 2019Adaptive Attention Span in Transformerscited by 148
  23. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  24. 2023Future Lens: Anticipating Subsequent Tokens from a Single Hidden Statecited by 116
  25. 2019Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challengecited by 86
  26. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  27. 2024Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recyclingcited by 48
  28. 2025Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potentialcited by 46
  29. 2025Apple Intelligence Foundation Language Models: Tech Report 2025cited by 25
  30. 2024TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devicescited by 9
  31. 2025SPD: Sync-Point Drop for Efficient Tensor Parallelism of Large Language Modelscited by 6
  32. 2024Communication Compression for Tensor Parallel LLM Inferencecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Speculative Speculative Decodingcited by 13

Similar, not cited

Close in wording, no citation either way

  1. 2025CARD: A Cache-Assisted Parallel Speculative Decoding Framework via Query-and-Correct Paradigm for Accelerating LLM Inference
  2. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices
  3. 2026BitNest: Bit-Nested Speculative Decoding for Memory-Efficient LLM Inference Acceleration
  4. 2025Tutorial Proposal: Speculative Decoding for Efficient LLM Inference
  5. 2025Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios
  6. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration
  7. 2025Speculative Verification: Exploiting Information Gain to Refine Speculative Decoding
  8. 2025Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decoding