# Speculation at a Distance: Where Edge-Cloud Speculative Decoding Actually Pays Off

arXiv: https://arxiv.org/abs/2606.25091 · HTML: https://mlsanity.com/paper/2606.25091/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving.
- Builds on 18 of its 21 references that are in the map.
- First posted on arXiv 23 Jun 2026.

## Placed in

- Efficient LLM inference and serving / [Early exit and edge inference](https://mlsanity.com/inference/early-exit-and-edge-inference/)

## Builds on (most cited first, 18 of 18)

- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding](https://mlsanity.com/paper/2401.07851/) (2024)
- [MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding](https://mlsanity.com/paper/2408.11049/) (2024)
- [TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput](https://mlsanity.com/paper/2406.14066/) (2024)
- [Decoding Speculative Decoding](https://mlsanity.com/paper/2402.01528/) (2024)
- [DSSD: Efficient Edge-Device LLM Deployment and Collaborative Inference via Distributed Split Speculative Decoding](https://mlsanity.com/paper/2507.12000/) (2025)
- [SLED: A Speculative LLM Decoding Framework for Efficient Edge Serving](https://mlsanity.com/paper/2506.09397/) (2025)
- [SpecEdge: Scalable Edge-Assisted Serving Framework for Interactive LLMs](https://mlsanity.com/paper/2505.17052/) (2025)
- [Distributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inference](https://mlsanity.com/paper/2405.14105/) (2024)
- [Collaborative Speculative Inference for Efficient LLM Inference Serving](https://mlsanity.com/paper/2503.10325/) (2025)
- [DSD: A Distributed Speculative Decoding Solution for Edge-Cloud Agile Large Model Serving](https://mlsanity.com/paper/2511.21669/) (2025)
- [Fast Collaborative Inference via Distributed Speculative Decoding](https://mlsanity.com/paper/2512.16273/) (2025)
- [PipeSD: An Efficient Cloud-Edge Collaborative Pipeline Inference Framework with Speculative Decoding](https://mlsanity.com/paper/2605.13319/) (2026)
- [A Pipelined Collaborative Speculative Decoding Framework for Efficient Edge-Cloud LLM Inference](https://mlsanity.com/paper/2603.19133/) (2026)
- [Revisiting the Arguments for Edge Computing Research](https://mlsanity.com/paper/2106.12224/) (2021)
- [Collaborative Large Language Model Inference via Resource-Aware Parallel Speculative Decoding](https://mlsanity.com/paper/2511.01695/) (2025)
- [A case study on latency, bandwidth and energy efficiency of mobile 5G and YouTube Edge service in London. Why the 5G ecosystem and energy efficiency matter?](https://mlsanity.com/paper/2310.14090/) (2023)

## Built on by (most cited first, 1 of 1)

- [AsymSpec: Efficient Cloud-Edge Speculative Decoding over Asymmetric Networks](https://mlsanity.com/paper/2608.04974/) (2026)

## Abstract (arXiv)

Speculative decoding (SD) accelerates LLM inference by 1.5-3 times when the draft and target models are co-located. This has motivated a distributed variant (DSD) that places the draft model on an edge device while the target stays in the cloud. We show with closed-form inequalities that DSD's per-request latency benefit is limited under WAN edge-cloud communication. If the server can host both models, co-located SD has lower latency and communication than synchronous DSD, with the same per-output FLOPs and model-weight memory. Pipelining can make DSD competitive with co-located SD only in low-RTT regimes where the round trip is shorter than the edge drafting time window; at WAN RTTs, the cloud round trip remains too large for pipelined DSD to beat co-located SD. Against cloud autoregressive decoding, DSD can reduce latency only inside a bounded window given the target-model speed, acceptance rate, and RTT. DSD is also infeasible against closed-source APIs without a verifier-only interface. The main case for DSD appears in multi-tenant capacity. Under cross-client overlap, offloading draft compute lets a saturated cloud server sustain (1 + γ,td/tv) times more concurrent clients at the same per-client rate, where γ is the speculation length and td, tv are the per-step draft and verification times. DSD should therefore be evaluated primarily by multi-tenant capacity and server throughput, not only by single-request latency.