mlsanityData to 7 Oct 2026

BALANCE: Hybrid Autoregressive-Speculative LLM Inference at the Network Edge

arXiv 2608.05926 ↗First posted 6 Aug 2026cs.NI cs.AIcited by 0

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Edge inference is a promising paradigm to provide large language model (LLM) inference services in next-generation mobile networks. LLM inference mainly relies on two approaches: Autoregressive decoding (AD) generates output tokens sequentially, resulting in long latency; Speculative decoding (SD) accelerates inference by using a small language model (SLM) to generate multiple draft tokens for LLM verification, but incurs extra memory costs. Due to this latency-memory tradeoff, neither approach alone can efficiently serve users with heterogeneous demands under limited edge computing resources. To address this challenge, we propose a hybrid autoregressive-speculative inference (BALANCE) framework for edge LLM inference. In BALANCE, an edge server hosts both an SLM and an LLM, admits users, assigns each admitted user to the AD or SD mode, and performs the two modes simultaneously. To maximize the number of served users, we formulate a task throughput maximization problem to jointly determine user admission and computing resource allocation between AD and SD under user latency requirements and server memory constraints. Since the problem is NP-hard, we develop a polynomial-time algorithm that transforms the original problem into two sub-problems and obtains a sub-optimal solution with a constant approximation guarantee. Experiments demonstrate that BALANCE consistently outperforms conventional AD and SD and significantly improves task throughput.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 38 of its 51 references that are in the map. First posted on arXiv 6 Aug 2026.

Builds on

38 of 51 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025When, What, and How: Rethinking Retrieval-Enhanced Speculative Decoding3 Nov
  2. 2025Jakiro: Boosting Speculative Decoding with Decoupled Multi-Head via MoE10 Feb
  3. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache5 Feb
  4. 2024A Theoretical Perspective for Speculative Decoding Algorithm30 Oct
  5. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration9 Oct
  6. 2024PEARL: Parallel Speculative Decoding with Adaptive Draft Length13 Aug
  7. 2024OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure25 Jun
  8. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices4 Jun
  9. 2024FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping5 Apr
  10. 2024Decoding Speculative Decoding2 Feb
  11. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan

Other references · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  3. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  4. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  5. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  6. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  7. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Predictioncited by 64
  8. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 43
  9. 2025Past-Future Scheduler for LLM Serving under SLA Guaranteescited by 20
  10. 2024D2-JSCC: Digital Deep Joint Source-channel Coding for Semantic Communicationscited by 18
  11. 2022iGniter: Interference-Aware GPU Resource Provisioning for Predictable DNN Inference in the Cloudcited by 13
  12. 2025SLED: A Speculative LLM Decoding Framework for Efficient Edge Servingcited by 13
15 more
  1. 2022Multi-user Co-inference with Batch Processing Capable Edge Servercited by 11
  2. 2024Resource Allocation in Large Language Model Integrated 6G Vehicular Networkscited by 11
  3. 2024TrimCaching: Parameter-sharing Edge Caching for AI Model Downloadingcited by 11
  4. 2025Edge-First Language Model Inference: Models, Metrics, and Tradeoffscited by 11
  5. 2025Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networkscited by 9
  6. 2025SlimCaching: Edge Caching of Mixture-of-Experts for Distributed Inferencecited by 7
  7. 2026Scheduling LLM Inference with Uncertainty-Aware Output Length Predictionscited by 7
  8. 2025Large Language Model Partitioning for Low-Latency Inference at the Edgecited by 5
  9. 2025PartialLoading: User Scheduling and Bandwidth Allocation for Parameter-sharing Edge Inferencecited by 4
  10. 2025Communication-Efficient Hybrid Language Model via Uncertainty-Aware Opportunistic and Compressed Transmissioncited by 4
  11. 2025Efficient LLM Inference over Heterogeneous Edge Networks with Speculative Decodingcited by 4
  12. 2023Graft: Efficient Inference Serving for Hybrid Deep Learning with SLO Guarantees via DNN Re-alignmentcited by 3
  13. 2026Network Edge Inference for Large Language Models: Principles, Techniques, and Opportunitiescited by 3
  14. 2025SLIDE: Simultaneous Model Downloading and Inference at the Wireless Network Edgecited by 2
  15. 2026SiftMoE: Similarity-Aware Energy-Efficient Expert Selection for Wireless Distributed MoE Inferencecited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Collaborative Large Language Model Inference via Resource-Aware Parallel Speculative Decoding
  2. 2025MNN-LLM: A Generic Inference Engine for Fast Large Language Model Deployment on Mobile Devices
  3. 2026Compiler-Assisted Speculative Sampling for Accelerated LLM Inference on Heterogeneous Edge Devices
  4. 2026Speculation at a Distance: Where Edge-Cloud Speculative Decoding Actually Pays Off
  5. 2026Lever: Speculative LLM Inference on Smartphones
  6. 2026A Pipelined Collaborative Speculative Decoding Framework for Efficient Edge-Cloud LLM Inference
  7. 2025Speculative Decoding in Decentralized LLM Inference: Turning Communication Latency into Computation Throughput
  8. 2026SpecFed: Accelerating Federated LLM Inference with Speculative Decoding and Compressed Transmission