Speculation at a Distance: Where Edge-Cloud Speculative Decoding Actually Pays Off
On the map: Efficient LLM inference and serving → Early exit and edge inference
Abstract (arXiv)
Speculative decoding (SD) accelerates LLM inference by 1.5-3 times when the draft and target models are co-located. This has motivated a distributed variant (DSD) that places the draft model on an edge device while the target stays in the cloud. We show with closed-form inequalities that DSD's per-request latency benefit is limited under WAN edge-cloud communication. If the server can host both models, co-located SD has lower latency and communication than synchronous DSD, with the same per-output FLOPs and model-weight memory. Pipelining can make DSD competitive with co-located SD only in low-RTT regimes where the round trip is shorter than the edge drafting time window; at WAN RTTs, the cloud round trip remains too large for pipelined DSD to beat co-located SD. Against cloud autoregressive decoding, DSD can reduce latency only inside a bounded window given the target-model speed, acceptance rate, and RTT. DSD is also infeasible against closed-source APIs without a verifier-only interface. The main case for DSD appears in multi-tenant capacity. Under cross-client overlap, offloading draft compute lets a saturated cloud server sustain (1 + γ,td/tv) times more concurrent clients at the same per-client rate, where γ is the speculation length and td, tv are the per-step draft and verification times. DSD should therefore be evaluated primarily by multi-tenant capacity and server throughput, not only by single-request latency.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 18 of its 21 references that are in the map. First posted on arXiv 23 Jun 2026.
Builds on
18 of 21 references.
Closest ancestors · 3
Same direction, newest first
- 2026PipeSD: An Efficient Cloud-Edge Collaborative Pipeline Inference Framework with Speculative Decoding13 May
- 2026A Pipelined Collaborative Speculative Decoding Framework for Efficient Edge-Cloud LLM Inference19 Mar
- 2025DSD: A Distributed Speculative Decoding Solution for Edge-Cloud Agile Large Model Serving26 Nov
Other references · 15
- 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
- 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
- 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 269
- 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decodingcited by 91
- 2024TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodputcited by 37
- 2024Decoding Speculative Decodingcited by 34
- 2025DSSD: Efficient Edge-Device LLM Deployment and Collaborative Inference via Distributed Split Speculative Decodingcited by 15
- 2025SLED: A Speculative LLM Decoding Framework for Efficient Edge Servingcited by 13
- 2025SpecEdge: Scalable Edge-Assisted Serving Framework for Interactive LLMscited by 10
- 2024Distributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inferencecited by 9
- 2025Collaborative Speculative Inference for Efficient LLM Inference Servingcited by 5
- 2025Fast Collaborative Inference via Distributed Speculative Decodingcited by 4
3 more
- 2021Revisiting the Arguments for Edge Computing Researchcited by 2
- 2025Collaborative Large Language Model Inference via Resource-Aware Parallel Speculative Decodingcited by 2
- 2023A case study on latency, bandwidth and energy efficiency of mobile 5G and YouTube Edge service in London. Why the 5G ecosystem and energy efficiency matter?cited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Speculative Decoding in Decentralized LLM Inference: Turning Communication Latency into Computation Throughput
- 2026Delay-Adaptive Speculation Control for Low-Latency Edge-Cloud LLM Inference
- 2026SPADE: Speculative Decoding for Precise and Low Cost Distributed Edge Cloud Inference
- 2026ConfigSpec: Profiling-Based Configuration Selection for Distributed Edge--Cloud Speculative LLM Serving
- 2026BALANCE: Hybrid Autoregressive-Speculative LLM Inference at the Network Edge
- 2026SpecFed: Accelerating Federated LLM Inference with Speculative Decoding and Compressed Transmission
- 2026FlexSpec: Frozen Drafts Meet Evolving Targets in Edge-Cloud Collaborative LLM Speculative Decoding
- 2026AceSpec: An Asymmetric Edge-Cloud Collaborative Framework for Communication-Efficient LLM Inference