# AMUSD: Asynchronous Multi-Device Speculative Decoding for LLM Acceleration

arXiv: https://arxiv.org/abs/2410.17375 · HTML: https://mlsanity.com/paper/2410.17375/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving.
- Builds on 16 of its 17 references that are in the map.
- First posted on arXiv 22 Oct 2024.

## Placed in

- Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/)

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://mlsanity.com/paper/2401.15077/) (2024)
- [SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification](https://mlsanity.com/paper/2305.09781/) (2023)
- [Blockwise Parallel Decoding for Deep Autoregressive Models](https://mlsanity.com/paper/1811.03115/) (2018)
- [Break the Sequential Dependency of LLM Inference Using Lookahead Decoding](https://mlsanity.com/paper/2402.02057/) (2024)
- [Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding](https://mlsanity.com/paper/2401.07851/) (2024)
- [Large Language Models for Software Engineering: Survey and Open Problems](https://mlsanity.com/paper/2310.03533/) (2023)
- [DistillSpec: Improving Speculative Decoding via Knowledge Distillation](https://mlsanity.com/paper/2310.08461/) (2023)
- [REST: Retrieval-Based Speculative Decoding](https://mlsanity.com/paper/2311.08252/) (2023)
- [Self-planning Code Generation with Large Language Models](https://mlsanity.com/paper/2303.06689/) (2023)
- [A Survey of Large Language Models for Code: Evolution, Benchmarking, and Future Trends](https://mlsanity.com/paper/2311.10372/) (2023)

## Built on by (most cited first, 14 of 14)

- [Speculative Decoding and Beyond: An In-Depth Survey of Techniques](https://mlsanity.com/paper/2502.19732/) (2025)
- [Speculative Speculative Decoding](https://mlsanity.com/paper/2603.03251/) (2026)
- [Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exits](https://mlsanity.com/paper/2505.21594/) (2025)
- [SpecEdge: Scalable Edge-Assisted Serving Framework for Interactive LLMs](https://mlsanity.com/paper/2505.17052/) (2025)
- [SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decoding](https://mlsanity.com/paper/2506.11309/) (2025)
- [DSVD: Dynamic Self-Verify Decoding for Faithful Generation in Large Language Models](https://mlsanity.com/paper/2503.03149/) (2025)
- [AHASD: Asynchronous Heterogeneous Architecture for LLM Adaptive Drafting Speculative Decoding on Mobile Devices](https://mlsanity.com/paper/2604.25326/) (2026)
- [When Parallel Drafter Meets Parallel Speculative Decoding](https://mlsanity.com/paper/2609.27396/) (2026)
- [FlowSpec: Continuous Pipelined Speculative Decoding for Efficient Distributed LLM Inference](https://mlsanity.com/paper/2507.02620/) (2025)
- [Speculative Sampling For Faster Molecular Dynamics](https://mlsanity.com/paper/2606.02455/) (2026)
- [LUMEN: Coordinated Failure Recovery for Distributed LLM Serving](https://mlsanity.com/paper/2606.17787/) (2026)
- [ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference](https://mlsanity.com/paper/2609.17943/) (2026)
- [AsymSpec: Efficient Cloud-Edge Speculative Decoding over Asymmetric Networks](https://mlsanity.com/paper/2608.04974/) (2026)
- [JetSpec++: Foresight Parallel Drafting with Self Refinements](https://mlsanity.com/paper/2610.19792/) (2026)

## Abstract (arXiv)

Large language models typically generate tokens autoregressively, using each token as input for the next. Recent work on Speculative Decoding has sought to accelerate this process by employing a smaller, faster draft model to more quickly generate candidate tokens. These candidates are then verified in parallel by the larger (original) verify model, resulting in overall speedup compared to using the larger model by itself in an autoregressive fashion. In this work, we introduce AMUSD (Asynchronous Multi-device Speculative Decoding), a system that further accelerates generation by decoupling the draft and verify phases into a continuous, asynchronous approach. Unlike conventional speculative decoding, where only one model (draft or verify) performs token generation at a time, AMUSD enables both models to perform predictions independently on separate devices (e.g., GPUs). We evaluate our approach over multiple datasets and show that AMUSD achieves an average 29% improvement over speculative decoding and up to 1.96× speedup over conventional autoregressive decoding, while achieving identical output quality. Our system is open-source and available at https://github.com/BradMcDanel/AMUSD/.