# SpecFed: Accelerating Federated LLM Inference with Speculative Decoding and Compressed Transmission

arXiv: https://arxiv.org/abs/2604.25777 · HTML: https://mlsanity.com/paper/2604.25777/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving.
- Builds on 18 of its 25 references that are in the map.
- First posted on arXiv 28 Apr 2026.

## Placed in

- Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/)

## Builds on (most cited first, 18 of 18)

- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [The Effect of Sampling Temperature on Problem Solving in Large Language Models](https://mlsanity.com/paper/2402.05201/) (2024)
- [Harnessing Multiple Large Language Models: A Survey on LLM Ensemble](https://mlsanity.com/paper/2502.18036/) (2025)
- [Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment](https://mlsanity.com/paper/2501.19309/) (2025)
- [Exploring the Impact of Temperature on Large Language Models:Hot or Cold?](https://mlsanity.com/paper/2506.07295/) (2025)
- [Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensembling](https://mlsanity.com/paper/2406.12585/) (2024)
- [Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensembling](https://mlsanity.com/paper/2410.03777/) (2024)
- [Uncertainty-Aware Hybrid Inference with On-Device Small and Remote Large Language Models](https://mlsanity.com/paper/2412.12687/) (2024)
- [DSSD: Efficient Edge-Device LLM Deployment and Collaborative Inference via Distributed Split Speculative Decoding](https://mlsanity.com/paper/2507.12000/) (2025)
- [Fast Large Language Model Collaborative Decoding via Speculation](https://mlsanity.com/paper/2502.01662/) (2025)
- [Over-the-Air Ensemble Inference with Model Privacy](https://mlsanity.com/paper/2202.03129/) (2022)
- [Decentralized Low-Latency Collaborative Inference via Ensembles on the Edge](https://mlsanity.com/paper/2206.03165/) (2022)
- [Quantize-Sample-and-Verify: LLM Acceleration via Adaptive Edge-Cloud Speculative Decoding](https://mlsanity.com/paper/2507.00605/) (2025)
- [Communication-Efficient Collaborative LLM Inference via Distributed Speculative Decoding](https://mlsanity.com/paper/2509.04576/) (2025)
- [Efficient LLM Inference over Heterogeneous Edge Networks with Speculative Decoding](https://mlsanity.com/paper/2510.11331/) (2025)
- [Fast Collaborative Inference via Distributed Speculative Decoding](https://mlsanity.com/paper/2512.16273/) (2025)
- [When to Ensemble: Identifying Token-Level Points for Stable and Fast LLM Ensembling](https://mlsanity.com/paper/2510.15346/) (2025)

## Abstract (arXiv)

Federated inference enhances LLM performance in edge computing through weighted averaging of distributed model predictions. However, autoregressive LLM inference requires frequent full-model forward passes across workers, severely limiting decoding throughput. Distributed deployment further aggravates this due to a communication bottleneck: each worker must transmit full token probability distributions per draft token, dominating end-to-end latency. To address these challenges, we introduce speculative decoding to enable parallel LLM processing and propose a top-K compressed transmission scheme with two server-side reconstruction strategies. We theoretically analyze the robustness of our method in terms of local reconstruction error, aggregation bias, and acceptance-rate bias, and derive corresponding bounds. Experiments demonstrate that our scheme achieves high generation fidelity while significantly reducing communication overhead.