# SPADE: Speculative Decoding for Precise and Low Cost Distributed Edge Cloud Inference

arXiv: https://arxiv.org/abs/2608.13076 · HTML: https://mlsanity.com/paper/2608.13076/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving.
- Builds on 14 of its 15 references that are in the map.
- First posted on arXiv 13 Aug 2026.

## Placed in

- Efficient LLM inference and serving / [Early exit and edge inference](https://mlsanity.com/inference/early-exit-and-edge-inference/)

## Builds on (most cited first, 14 of 14)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding](https://mlsanity.com/paper/2404.16710/) (2024)
- [Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding](https://mlsanity.com/paper/2401.07851/) (2024)
- [I-BERT: Integer-only BERT Quantization](https://mlsanity.com/paper/2101.01321/) (2021)
- [Neural Compression and Filtering for Edge-assisted Real-time Object Detection in Challenged Networks](https://mlsanity.com/paper/2007.15818/) (2020)
- [SplitEE: Early Exit in Deep Neural Networks with Split Computing](https://mlsanity.com/paper/2309.09195/) (2023)
- [CEEBERT: Cross-Domain Inference in Early Exit BERT](https://mlsanity.com/paper/2405.15039/) (2024)
- [I-SplitEE: Image classification in Split Computing DNNs with Early Exits](https://mlsanity.com/paper/2401.10541/) (2024)
- [Distributed Inference on Mobile Edge and Cloud: An Early Exit based Clustering Approach](https://mlsanity.com/paper/2410.05338/) (2024)

## Abstract (arXiv)

Large Language Models (LLMs) have achieved remarkable success in natural language understanding and generation, but their deployment is constrained by high computational demands. Deploying smaller LLMs directly on the edge can circumvent this, but with degraded accuracy. Deploying smaller cloud-based big LLMs preserves performance, but at the cost of expensive per-token computation. We present a distributed inference framework, our, that integrates speculative decoding (SD) across edge and cloud. A compact draft model deployed on the edge generates candidate tokens rapidly, and a large verifier model on the cloud validates these tokens in parallel. Accepted tokens are retained, while only rejections trigger verifier correction, substantially reducing the number of cloud queries. Our plug-and-play design shifts the bulk of computation to the edge, significantly lowers inference time and cloud cost, and preserves the accuracy of the big model without any retraining requirement. Our approach demonstrates a practical path toward scalable, cost-efficient, and accurate deployment of LLMs in real-world environments. Experimental results across multiple Natural Language Processing tasks using SpecBench and CNN/Dailymail datasets demonstrate that our reduces the cloud model calls by 76% with zero loss in accuracy as compared to the full model.