# BALANCE: Hybrid Autoregressive-Speculative LLM Inference at the Network Edge

arXiv: https://arxiv.org/abs/2608.05926 · HTML: https://mlsanity.com/paper/2608.05926/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving.
- Builds on 38 of its 51 references that are in the map.
- First posted on arXiv 6 Aug 2026.

## Placed in

- Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/)

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU](https://mlsanity.com/paper/2303.06865/) (2023)
- [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://mlsanity.com/paper/2403.02310/) (2024)
- [Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing](https://mlsanity.com/paper/2404.14618/) (2024)
- [Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding](https://mlsanity.com/paper/2401.07851/) (2024)
- [Mobile Edge Intelligence for Large Language Models: A Contemporary Survey](https://mlsanity.com/paper/2407.18921/) (2024)
- [PEARL: Parallel Speculative Decoding with Adaptive Draft Length](https://mlsanity.com/paper/2408.11850/) (2024)
- [SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration](https://mlsanity.com/paper/2410.06916/) (2024)
- [OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure](https://mlsanity.com/paper/2406.17276/) (2024)
- [Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction](https://mlsanity.com/paper/2404.08509/) (2024)
- [SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices](https://mlsanity.com/paper/2406.02532/) (2024)
- [ExeGPT: Constraint-Aware Resource Scheduling for LLM Inference](https://mlsanity.com/paper/2404.07947/) (2024)
- [Decoding Speculative Decoding](https://mlsanity.com/paper/2402.01528/) (2024)
- [A Theoretical Perspective for Speculative Decoding Algorithm](https://mlsanity.com/paper/2411.00841/) (2024)
- [QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache](https://mlsanity.com/paper/2502.10424/) (2025)
- [FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping](https://mlsanity.com/paper/2404.03865/) (2024)
- [D2-JSCC: Digital Deep Joint Source-channel Coding for Semantic Communications](https://mlsanity.com/paper/2403.07338/) (2024)
- [Past-Future Scheduler for LLM Serving under SLA Guarantees](https://mlsanity.com/paper/2507.10150/) (2025)
- [iGniter: Interference-Aware GPU Resource Provisioning for Predictable DNN Inference in the Cloud](https://mlsanity.com/paper/2211.01713/) (2022)
- [Edge-First Language Model Inference: Models, Metrics, and Tradeoffs](https://mlsanity.com/paper/2505.16508/) (2025)
- [SLED: A Speculative LLM Decoding Framework for Efficient Edge Serving](https://mlsanity.com/paper/2506.09397/) (2025)
- [Multi-user Co-inference with Batch Processing Capable Edge Server](https://mlsanity.com/paper/2206.06304/) (2022)
- [Resource Allocation in Large Language Model Integrated 6G Vehicular Networks](https://mlsanity.com/paper/2403.19016/) (2024)
- [TrimCaching: Parameter-sharing Edge Caching for AI Model Downloading](https://mlsanity.com/paper/2404.14204/) (2024)
- [Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks](https://mlsanity.com/paper/2503.14882/) (2025)
- [Scheduling LLM Inference with Uncertainty-Aware Output Length Predictions](https://mlsanity.com/paper/2604.00499/) (2026)
- [SlimCaching: Edge Caching of Mixture-of-Experts for Distributed Inference](https://mlsanity.com/paper/2507.06567/) (2025)
- [Jakiro: Boosting Speculative Decoding with Decoupled Multi-Head via MoE](https://mlsanity.com/paper/2502.06282/) (2025)
- [Large Language Model Partitioning for Low-Latency Inference at the Edge](https://mlsanity.com/paper/2505.02533/) (2025)

## Abstract (arXiv)

Edge inference is a promising paradigm to provide large language model (LLM) inference services in next-generation mobile networks. LLM inference mainly relies on two approaches: Autoregressive decoding (AD) generates output tokens sequentially, resulting in long latency; Speculative decoding (SD) accelerates inference by using a small language model (SLM) to generate multiple draft tokens for LLM verification, but incurs extra memory costs. Due to this latency-memory tradeoff, neither approach alone can efficiently serve users with heterogeneous demands under limited edge computing resources. To address this challenge, we propose a hybrid autoregressive-speculative inference (BALANCE) framework for edge LLM inference. In BALANCE, an edge server hosts both an SLM and an LLM, admits users, assigns each admitted user to the AD or SD mode, and performs the two modes simultaneously. To maximize the number of served users, we formulate a task throughput maximization problem to jointly determine user admission and computing resource allocation between AD and SD under user latency requirements and server memory constraints. Since the problem is NP-hard, we develop a polynomial-time algorithm that transforms the original problem into two sub-problems and obtains a sub-optimal solution with a constant approximation guarantee. Experiments demonstrate that BALANCE consistently outperforms conventional AD and SD and significantly improves task throughput.