# Quantize-Sample-and-Verify: LLM Acceleration via Adaptive Edge-Cloud Speculative Decoding

arXiv: https://arxiv.org/abs/2507.00605 · HTML: https://mlsanity.com/paper/2507.00605/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 22 references that are in the map.
- First posted on arXiv 1 Jul 2025.

## Builds on (most cited first, 13 of 13)

- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification](https://mlsanity.com/paper/2405.14256/) (2024)
- [AI Flow at the Network Edge](https://mlsanity.com/paper/2411.12469/) (2024)
- [Uncertainty-Aware Hybrid Inference with On-Device Small and Remote Large Language Models](https://mlsanity.com/paper/2412.12687/) (2024)
- [Vector Quantization Prompting for Continual Learning](https://mlsanity.com/paper/2410.20444/) (2024)
- [Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exits](https://mlsanity.com/paper/2505.21594/) (2025)
- [Quantized Prompt for Efficient Generalization of Vision-Language Models](https://mlsanity.com/paper/2407.10704/) (2024)
- [Latency-Distortion Tradeoffs in Communicating Classification Results over Noisy Channels](https://mlsanity.com/paper/2404.14586/) (2024)
- [Multi-RAT for IoT: The Potential in Combining LoRaWAN and NB-IoT](https://mlsanity.com/paper/2104.10536/) (2021)

## Built on by (most cited first, 4 of 4)

- [CIAR: Interval-based Collaborative Decoding for Image Generation Acceleration](https://mlsanity.com/paper/2603.25463/) (2026)
- [SpecFed: Accelerating Federated LLM Inference with Speculative Decoding and Compressed Transmission](https://mlsanity.com/paper/2604.25777/) (2026)
- [Multi-SPIN: Multi-Access Speculative Inference for Cooperative Token Generation at the Edge](https://mlsanity.com/paper/2606.04581/) (2026)
- [AsymSpec: Efficient Cloud-Edge Speculative Decoding over Asymmetric Networks](https://mlsanity.com/paper/2608.04974/) (2026)

## Abstract (arXiv)

In edge-cloud speculative decoding (SD), edge devices equipped with small language models (SLMs) generate draft tokens that are verified by large language models (LLMs) in the cloud. A key bottleneck in such systems is the limited communication bandwidth between edge and cloud, which necessitates quantization of the information transmitted about generated tokens. In this work, we introduce a novel quantize-sample (Q-S) strategy that provably preserves the output distribution of the cloud-based model, ensuring that the verified tokens match the distribution of those that would have been generated directly by the LLM. We develop a throughput model for edge-cloud SD that explicitly accounts for communication latency. Leveraging this model, we propose an adaptive mechanism that optimizes token throughput by dynamically adjusting the draft length and quantization precision in response to both semantic uncertainty and channel conditions. Simulations demonstrate that the proposed Q-S approach significantly improves decoding efficiency in realistic edge-cloud deployment scenarios.