mlsanityData to 14 Oct 2026

SpecEdge: Scalable Edge-Assisted Serving Framework for Interactive LLMs

arXiv 2505.17052 ↗First posted 16 May 2025cs.CL cs.AIcited by 10

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large language models (LLMs) power many modern applications, but serving them at scale remains costly and resource-intensive. Current server-centric systems overlook consumer-grade GPUs at the edge. We introduce SpecEdge, an edge-assisted inference framework that splits LLM workloads between edge and server GPUs using a speculative decoding scheme, exchanging only token outputs over the network. SpecEdge employs proactive edge drafting to overlap edge token creation with server verification and pipeline-aware scheduling that interleaves multiple user requests to increase server-side throughput. Experiments show SpecEdge enhances overall cost efficiency by 1.91x through achieving 2.22x server throughput, and reduces inter token latency by 11.24% compared to a server-only baseline, introducing a scalable, cost-effective paradigm for LLM serving. The code is available at https://github.com/kaist-ina/specedge

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 41 of its 53 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

41 of 53 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  9. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  11. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  12. 2023Mistral 7Bcited by 3,164
29 more
  1. 2016Pointer Sentinel Mixture Modelscited by 3,096
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  3. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  4. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  5. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  6. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  7. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  8. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  9. 2022Efficiently Scaling Transformer Inferencecited by 539
  10. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  11. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  12. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  13. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  14. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  15. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 269
  16. 2019Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computingcited by 228
  17. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  18. 2023SpecTr: Fast Speculative Decoding via Optimal Transportcited by 129
  19. 2019Convergence of Edge Computing and Deep Learning: A Comprehensive Surveycited by 120
  20. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingcited by 103
  21. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 103
  22. 2024PEARL: Parallel Speculative Decoding with Adaptive Draft Lengthcited by 85
  23. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devicescited by 46
  24. 2023Distributed Inference and Fine-tuning of Large Language Models Over The Internetcited by 42
  25. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Modelscited by 38
  26. 2020Training Large Neural Networks with Constant Memory using a New Execution Algorithmcited by 34
  27. 2024AMUSD: Asynchronous Multi-Device Speculative Decoding for LLM Accelerationcited by 14
  28. 2024The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculationcited by 14
  29. 2024Distributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inferencecited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 8.

10 papers.

  1. 2025Fast LLM Post-training via Decoupled and Fastest-of-N Speculationcited by 4
  2. 2026PipeSD: An Efficient Cloud-Edge Collaborative Pipeline Inference Framework with Speculative Decodingcited by 4
  3. 2026Speculation at a Distance: Where Edge-Cloud Speculative Decoding Actually Pays Offcited by 1
  4. 2025CoEdge-RAG: Optimizing Hierarchical Scheduling for Retrieval-Augmented LLMs in Collaborative Edge Computingcited by 0
  5. 2026Embedded AI Companion System on Edge Devicescited by 0
  6. 2026CoVStream: Edge-Cloud Collaboration for Understanding of Long Video Streamscited by 0
  7. 2026LASER: Load-Aware Serving with Early-Exit for Reasoning LLMs at the Edgecited by 0
  8. 2026AsymSpec: Efficient Cloud-Edge Speculative Decoding over Asymmetric Networkscited by 0
  9. 2026EMMI: Edge Multi-Modal Intelligence for Communication-Efficient MLLM Inference via Fused Representation Compressioncited by 0
  10. 2026CIDERS: Cloud-Edge LLM Collaborative Learning via Accelerating Personalized Bilevel Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A Measurement Study of LLM Inference Trade-offs Across Edge Continuum Hardware
  2. 2026WISP: Waste- and Interference-Suppressed Distributed Speculative LLM Serving at the Edge via Dynamic Drafting and SLO-Aware Batching
  3. 2026ConfigSpec: Profiling-Based Configuration Selection for Distributed Edge--Cloud Speculative LLM Serving
  4. 2025Parallax: Efficient LLM Inference Service over Decentralized Environment
  5. 2026AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System
  6. 2026WANSpec: Leveraging Global Compute Capacity for LLM Inference
  7. 2026Compiler-Assisted Speculative Sampling for Accelerated LLM Inference on Heterogeneous Edge Devices
  8. 2025SLED: A Speculative LLM Decoding Framework for Efficient Edge Serving