mlsanityData to 7 Oct 2026

AMUSD: Asynchronous Multi-Device Speculative Decoding for LLM Acceleration

arXiv 2410.17375 ↗First posted 22 Oct 2024cs.CL cs.DC cs.LGcited by 13

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Large language models typically generate tokens autoregressively, using each token as input for the next. Recent work on Speculative Decoding has sought to accelerate this process by employing a smaller, faster draft model to more quickly generate candidate tokens. These candidates are then verified in parallel by the larger (original) verify model, resulting in overall speedup compared to using the larger model by itself in an autoregressive fashion. In this work, we introduce AMUSD (Asynchronous Multi-device Speculative Decoding), a system that further accelerates generation by decoupling the draft and verify phases into a continuous, asynchronous approach. Unlike conventional speculative decoding, where only one model (draft or verify) performs token generation at a time, AMUSD enables both models to perform predictions independently on separate devices (e.g., GPUs). We evaluate our approach over multiple datasets and show that AMUSD achieves an average 29% improvement over speculative decoding and up to 1.96× speedup over conventional autoregressive decoding, while achieving identical output quality. Our system is open-source and available at https://github.com/BradMcDanel/AMUSD/.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 16 of its 17 references that are in the map. First posted on arXiv 22 Oct 2024.

Builds on

16 of 17 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  2. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  3. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  4. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  5. 2023REST: Retrieval-Based Speculative Decoding14 Nov
  6. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  7. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May

Other references · 9

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  6. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  7. 2023Large Language Models for Software Engineering: Survey and Open Problemscited by 178
  8. 2023Self-planning Code Generation with Large Language Modelscited by 121
  9. 2023A Survey of Large Language Models for Code: Evolution, Benchmarking, and Future Trendscited by 66

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 7.

13 papers.

Sort:

By area:

  1. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
  2. 2026Speculative Speculative Decodingcited by 13
  3. 2025SpecEdge: Scalable Edge-Assisted Serving Framework for Interactive LLMscited by 10
  4. 2025SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decodingcited by 9
  5. 2025Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exitscited by 8
  6. 2025DSVD: Dynamic Self-Verify Decoding for Faithful Generation in Large Language Modelscited by 2
  7. 2025FlowSpec: Continuous Pipelined Speculative Decoding for Efficient Distributed LLM Inferencecited by 1
  8. 2026AHASD: Asynchronous Heterogeneous Architecture for LLM Adaptive Drafting Speculative Decoding on Mobile Devicescited by 1
  9. 2026Speculative Sampling For Faster Molecular Dynamicscited by 1
  10. 2026LUMEN: Coordinated Failure Recovery for Distributed LLM Servingcited by 1
  11. 2026AsymSpec: Efficient Cloud-Edge Speculative Decoding over Asymmetric Networkscited by 0
  12. 2026ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inferencecited by 0
1 more
  1. 2026When Parallel Drafter Meets Parallel Speculative Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices
  2. 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference
  3. 2023Accelerating LLM Inference with Staged Speculative Decoding
  4. 2025DuoDecoding: Hardware-aware Heterogeneous Speculative Decoding with Dynamic Multi-Sequence Drafting
  5. 2025Speculative Verification: Exploiting Information Gain to Refine Speculative Decoding
  6. 2026Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware
  7. 2025Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios
  8. 2025Tutorial Proposal: Speculative Decoding for Efficient LLM Inference