mlsanityData to 7 Oct 2026

FlexSpec: Frozen Drafts Meet Evolving Targets in Edge-Cloud Collaborative LLM Speculative Decoding

arXiv 2601.00644 ↗First posted 2 Jan 2026cs.DCcited by 3

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

Deploying large language models (LLMs) in mobile and edge computing environments is constrained by limited on-device resources, scarce wireless bandwidth, and frequent model evolution. Although edge-cloud collaborative inference with speculative decoding (SD) can reduce end-to-end latency by executing a lightweight draft model at the edge and verifying it with a cloud-side target model, existing frameworks fundamentally rely on tight coupling between the two models. Consequently, repeated model synchronization introduces excessive communication overhead, increasing end-to-end latency, and ultimately limiting the scalability of SD in edge environments. To address these limitations, we propose FlexSpec, a communication-efficient collaborative inference framework tailored for evolving edge-cloud systems. The core design of FlexSpec is a shared-backbone architecture that allows a single and static edge-side draft model to remain compatible with a large family of evolving cloud-side target models. By decoupling edge deployment from cloud-side model updates, FlexSpec eliminates the need for edge-side retraining or repeated model downloads, substantially reducing communication and maintenance costs. Furthermore, to accommodate time-varying wireless conditions and heterogeneous device constraints, we develop a channel-aware adaptive speculation mechanism that dynamically adjusts the speculative draft length based on real-time channel state information and device energy budgets. Extensive experiments demonstrate that FlexSpec achieves superior performance compared to conventional SD approaches in terms of inference efficiency.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 32 of its 44 references that are in the map. First posted on arXiv 2 Jan 2026.

Builds on

32 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges8 Mar

Other references · 31

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  7. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  8. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  9. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  10. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  11. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  12. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
19 more
  1. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  2. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  3. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
  4. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 178
  5. 2023REST: Retrieval-Based Speculative Decodingcited by 148
  6. 2020SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloudcited by 109
  7. 2024DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Modelscited by 98
  8. 2018Towards an Intelligent Edge: Wireless Communication Meets Machine Learningcited by 94
  9. 2018Wireless Network Intelligence at the Edgecited by 90
  10. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 84
  11. 2023Inference with Reference: Lossless Acceleration of Large Language Modelscited by 63
  12. 2024When Search Engine Services meet Large Language Models: Visions and Challengescited by 59
  13. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 58
  14. 2023Lookahead: An Inference Acceleration Framework for Large Language Model with Lossless Generation Accuracycited by 37
  15. 2025Towards AI Search Paradigmcited by 19
  16. 2023Semantic Communications for Image Recovery and Classification via Deep Joint Source and Channel Codingcited by 18
  17. 2025DSSD: Efficient Edge-Device LLM Deployment and Collaborative Inference via Distributed Split Speculative Decodingcited by 15
  18. 2025Edge-Cloud Collaborative Computing on Distributed Intelligence and Model Optimization: A Surveycited by 13
  19. 2025The Larger the Merrier? Efficient Large AI Model Inference in Wireless Edge Networkscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026UniCreative: Unifying Long-form Logic and Short-form Sparkle via Reference-Free Reinforcement Learningcited by 0
  2. 2026Delay-Adaptive Speculation Control for Low-Latency Edge-Cloud LLM Inferencecited by 0
  3. 2026SWRouter: Similarity-Contractive Window Routing for Multi-Turn Large Language Model Conversationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A Pipelined Collaborative Speculative Decoding Framework for Efficient Edge-Cloud LLM Inference
  2. 2026AceSpec: An Asymmetric Edge-Cloud Collaborative Framework for Communication-Efficient LLM Inference
  3. 2025Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges
  4. 2026ConfigSpec: Profiling-Based Configuration Selection for Distributed Edge--Cloud Speculative LLM Serving
  5. 2026SPADE: Speculative Decoding for Precise and Low Cost Distributed Edge Cloud Inference
  6. 2025DSD: A Distributed Speculative Decoding Solution for Edge-Cloud Agile Large Model Serving
  7. 2026Speculation at a Distance: Where Edge-Cloud Speculative Decoding Actually Pays Off
  8. 2025Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exits