# Conformal Sparsification for Bandwidth-Efficient Edge-Cloud Speculative Decoding

arXiv: https://arxiv.org/abs/2510.09942 · HTML: https://mlsanity.com/paper/2510.09942/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving.
- Builds on 16 of its 21 references that are in the map.
- First posted on arXiv 11 Oct 2025.

## Placed in

- Efficient LLM inference and serving / [Early exit and edge inference](https://mlsanity.com/inference/early-exit-and-edge-inference/)

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Adaptive Conformal Inference Under Distribution Shift](https://mlsanity.com/paper/2106.00170/) (2021)
- [One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling](https://mlsanity.com/paper/1312.3005/) (2013)
- [Model compression via distillation and quantization](https://mlsanity.com/paper/1802.05668/) (2018)
- [Truncation Sampling as Language Model Desmoothing](https://mlsanity.com/paper/2210.15191/) (2022)
- [Theoretical Foundations of Conformal Prediction](https://mlsanity.com/paper/2411.11824/) (2024)
- [Conformal PID Control for Time Series Prediction](https://mlsanity.com/paper/2307.16895/) (2023)
- [Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding](https://mlsanity.com/paper/2502.01563/) (2025)
- [A Theoretical Perspective for Speculative Decoding Algorithm](https://mlsanity.com/paper/2411.00841/) (2024)
- [Uncertainty-Aware Hybrid Inference with On-Device Small and Remote Large Language Models](https://mlsanity.com/paper/2412.12687/) (2024)
- [Localized Adaptive Risk Control](https://mlsanity.com/paper/2405.07976/) (2024)
- [Foundations of Top-k Decoding For Language Models](https://mlsanity.com/paper/2505.19371/) (2025)
- [Latency-Distortion Tradeoffs in Communicating Classification Results over Noisy Channels](https://mlsanity.com/paper/2404.14586/) (2024)

## Built on by (most cited first, 1 of 1)

- [AsymSpec: Efficient Cloud-Edge Speculative Decoding over Asymmetric Networks](https://mlsanity.com/paper/2608.04974/) (2026)

## Abstract (arXiv)

Edge-cloud speculative decoding (SD) accelerates inference by having a cloud-based large language model (LLM) that verifies draft tokens generated by a resource-constrained small language model (SLM) at the edge. A central bottleneck is the limited bandwidth of the edge-cloud link, which necessitates efficient compression of draft token distributions. We first derive an information-theoretic bound that decomposes the token rejection rate into contributions from SLM-LLM distribution mismatch and from quantization distortion. Guided by this analysis, we propose the Sparse Quantize-and-Sample SD (SQS-SD) framework, which exploits distributional sparsity through structured sparsification and lattice-based quantization. Within this framework, K-SQS applies fixed top-K truncation, while C-SQS adaptively adjusts the retained token set via online conformal prediction to ensure bounded deviation from the dense distribution. Empirical results confirm that both approaches improve end-to-end latency and rejection rates in complimentary operating regimes.