mlsanityData to 14 Oct 2026

Conformal Sparsification for Bandwidth-Efficient Edge-Cloud Speculative Decoding

arXiv 2510.09942 ↗First posted 11 Oct 2025cs.LG cs.AI cs.ITcited by 1

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

Edge-cloud speculative decoding (SD) accelerates inference by having a cloud-based large language model (LLM) that verifies draft tokens generated by a resource-constrained small language model (SLM) at the edge. A central bottleneck is the limited bandwidth of the edge-cloud link, which necessitates efficient compression of draft token distributions. We first derive an information-theoretic bound that decomposes the token rejection rate into contributions from SLM-LLM distribution mismatch and from quantization distortion. Guided by this analysis, we propose the Sparse Quantize-and-Sample SD (SQS-SD) framework, which exploits distributional sparsity through structured sparsification and lattice-based quantization. Within this framework, K-SQS applies fixed top-K truncation, while C-SQS adaptively adjusts the retained token set via online conformal prediction to ensure bounded deviation from the dense distribution. Empirical results confirm that both approaches improve end-to-end latency and rejection rates in complimentary operating regimes.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 16 of its 21 references that are in the map. First posted on arXiv 11 Oct 2025.

Builds on

16 of 21 references.

References · 16

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  3. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  4. 2018Hierarchical Neural Story Generationcited by 1,193
  5. 2021Adaptive Conformal Inference Under Distribution Shiftcited by 375
  6. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  7. 2018Model compression via distillation and quantizationcited by 289
  8. 2022Truncation Sampling as Language Model Desmoothingcited by 123
  9. 2024Theoretical Foundations of Conformal Predictioncited by 101
  10. 2023Conformal PID Control for Time Series Predictioncited by 88
  11. 2025Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understandingcited by 47
  12. 2024A Theoretical Perspective for Speculative Decoding Algorithmcited by 29
4 more
  1. 2024Uncertainty-Aware Hybrid Inference with On-Device Small and Remote Large Language Modelscited by 19
  2. 2024Localized Adaptive Risk Controlcited by 17
  3. 2025Foundations of Top-k Decoding For Language Modelscited by 10
  4. 2024Latency-Distortion Tradeoffs in Communicating Classification Results over Noisy Channelscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026AsymSpec: Efficient Cloud-Edge Speculative Decoding over Asymmetric Networkscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Quantize-Sample-and-Verify: LLM Acceleration via Adaptive Edge-Cloud Speculative Decoding
  2. 2026SPADE: Speculative Decoding for Precise and Low Cost Distributed Edge Cloud Inference
  3. 2026A Pipelined Collaborative Speculative Decoding Framework for Efficient Edge-Cloud LLM Inference
  4. 2026Delay-Adaptive Speculation Control for Low-Latency Edge-Cloud LLM Inference
  5. 2026Speculation at a Distance: Where Edge-Cloud Speculative Decoding Actually Pays Off
  6. 2026TSS: Target-Side Sparsification for Speculative Decoding in Domain-Specific Large Language Models
  7. 2025SLED: A Speculative LLM Decoding Framework for Efficient Edge Serving
  8. 2025DSD: A Distributed Speculative Decoding Solution for Edge-Cloud Agile Large Model Serving