mlsanityData to 7 Oct 2026

WISP: Waste- and Interference-Suppressed Distributed Speculative LLM Serving at the Edge via Dynamic Drafting and SLO-Aware Batching

arXiv 2601.11652 ↗First posted 15 Jan 2026cs.DC cs.AIcited by 1

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

As Large Language Models (LLMs) become increasingly accessible to end users, an ever-growing number of inference requests are initiated from edge devices and computed on centralized GPU clusters. However, the resulting exponential growth in computation workload is placing significant strain on data centers, while edge devices remain largely underutilized, leading to imbalanced workloads and resource inefficiency across the network. Integrating edge devices into the LLM inference process via speculative decoding helps balance the workload between the edge and the cloud, while maintaining lossless prediction accuracy. In this paper, we identify and formalize two critical bottlenecks that limit the efficiency and scalability of distributed speculative LLM serving: Wasted Drafting Time and Verification Interference. To address these challenges, we propose WISP, an efficient and SLO-aware distributed LLM inference system that consists of an intelligent speculation controller, a verification time estimator, and a verification batch scheduler. These components collaboratively enhance drafting efficiency and optimize verification request scheduling on the server. Extensive numerical results show that WISP improves system capacity by up to 2.1x and 4.1x, and increases system goodput by up to 1.94x and 3.7x, compared to centralized serving and SLED, respectively.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 41 references that are in the map. First posted on arXiv 15 Jan 2026.

Builds on

23 of 41 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  8. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  9. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
  10. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  11. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  12. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 101
10 more
  1. 2024EDGE-LLM: Enabling Efficient Large Language Model Adaptation on Edge Devices via Layerwise Unified Compression and Adaptive Layer Tuning and Votingcited by 26
  2. 2024Galaxy: A Resource-Efficient Collaborative Edge AI System for In-situ Transformer Inferencecited by 25
  3. 2025DSSD: Efficient Edge-Device LLM Deployment and Collaborative Inference via Distributed Split Speculative Decodingcited by 15
  4. 2025SLED: A Speculative LLM Decoding Framework for Efficient Edge Servingcited by 13
  5. 2024SplitLLM: Collaborative Inference of LLMs for Model Placement and Throughput Optimizationcited by 10
  6. 2022All You Need Is Logs: Improving Code Completion by Learning from Anonymous IDE Usage Logscited by 8
  7. 2025Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exitscited by 8
  8. 2025Optimal Scheduling Algorithms for LLM Inference: Theory and Practicecited by 5
  9. 2025DSD: A Distributed Speculative Decoding Solution for Edge-Cloud Agile Large Model Servingcited by 5
  10. 2025P/D-Device: Disaggregated Large Language Model between Cloud and Devicescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026AsymSpec: Efficient Cloud-Edge Speculative Decoding over Asymmetric Networkscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ConfigSpec: Profiling-Based Configuration Selection for Distributed Edge--Cloud Speculative LLM Serving
  2. 2026Compiler-Assisted Speculative Sampling for Accelerated LLM Inference on Heterogeneous Edge Devices
  3. 2025SpecEdge: Scalable Edge-Assisted Serving Framework for Interactive LLMs
  4. 2026WANSpec: Leveraging Global Compute Capacity for LLM Inference
  5. 2025FlowSpec: Continuous Pipelined Speculative Decoding for Efficient Distributed LLM Inference
  6. 2026SpecFed: Accelerating Federated LLM Inference with Speculative Decoding and Compressed Transmission
  7. 2025Parallax: Efficient LLM Inference Service over Decentralized Environment
  8. 2025Collaborative Speculative Inference for Efficient LLM Inference Serving