mlsanityData to 13 Oct 2026

GoodSpeed: Optimizing Fair Goodput with Adaptive Speculative Decoding in Distributed Edge Inference

arXiv 2512.09963 ↗First posted 10 Dec 2025cs.DCcited by 0

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Large language models (LLMs) have revolutionized natural language processing, yet their high computational demands pose significant challenges for real-time inference, especially in multi-user server speculative decoding and resource-constrained environments. Speculative decoding has emerged as a promising technique to accelerate LLM inference by using lightweight draft models to generate candidate tokens, which are subsequently verified by a larger, more accurate model. However, ensuring both high goodput (the effective rate of accepted tokens) and fairness across multiple draft servers cooperating with a central verification server remains an open challenge. This paper introduces GOODSPEED, a novel distributed inference framework that optimizes goodput through adaptive speculative decoding. GOODSPEED employs a central verification server that coordinates a set of heterogeneous draft servers, each running a small language model to generate speculative tokens. To manage resource allocation effectively, GOODSPEED incorporates a gradient scheduling algorithm that dynamically assigns token verification tasks, maximizing a logarithmic utility function to ensure proportional fairness across servers. By processing speculative outputs from all draft servers in parallel, the framework enables efficient collaboration between the verification server and distributed draft generators, streamlining both latency and throughput. Through rigorous fluid sample path analysis, we show that GOODSPEED converges to the optimal goodput allocation in steady-state conditions and maintains near-optimal performance with provably bounded error under dynamic workloads. These results demonstrate that GOODSPEED provides a scalable, fair and efficient solution for multi-server speculative decoding in distributed LLM inference systems.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 32 references that are in the map. First posted on arXiv 10 Dec 2025.

Builds on

23 of 32 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025SPIN: Accelerating Large Language Model Inference with Heterogeneous Speculative Models20 Mar
  2. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration9 Oct
  3. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  4. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  5. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  6. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  7. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  8. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 15

  1. 2017Attention Is All You Needcited by 60,106
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,831
  3. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 602
  4. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 545
  5. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 514
  6. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 452
  7. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 258
  8. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  9. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  10. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 78
  11. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 54
  12. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 45
3 more
  1. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 30
  2. 2025DSSD: Efficient Edge-Device LLM Deployment and Collaborative Inference via Distributed Split Speculative Decodingcited by 15
  3. 2025SLED: A Speculative LLM Decoding Framework for Efficient Edge Servingcited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Speculative Decoding in Decentralized LLM Inference: Turning Communication Latency into Computation Throughput
  2. 2026SPADE: Speculative Decoding for Precise and Low Cost Distributed Edge Cloud Inference
  3. 2025FlowSpec: Continuous Pipelined Speculative Decoding for Efficient Distributed LLM Inference
  4. 2025Collaborative Speculative Inference for Efficient LLM Inference Serving
  5. 2025Speculative Decoding: Performance or Illusion?
  6. 2024Minions: Accelerating Large Language Model Inference with Aggregated Speculative Execution
  7. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding
  8. 2025Fail Fast, Win Big: Rethinking the Drafting Strategy in Speculative Decoding via Diffusion LLMs