mlsanityData to 7 Oct 2026

PICE: A Semantic-Driven Progressive Inference System for LLM Serving in Cloud-Edge Networks

arXiv 2501.09367 ↗First posted 16 Jan 2025cs.DCcited by 5

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

Large language models (LLMs), while driving a new wave of interactive AI applications across numerous domains, suffer from high inference costs and heavy cloud dependency. Motivated by the redundancy phenomenon in linguistics, we propose a progressive inference paradigm over cloud and edge, i.e., firstly generating the sketch of the answer by LLMs at cloud, and then conducting parallel extension to fill in details by small models (SLMs) at edge. Progressive inference offers potential benefits to improve throughput and reduce inference latency while facing key implementation challenges, including decreased response quality from SLMs, a tradeoff between the brevity and comprehensiveness of sketches, as well as increased latency caused by network transmission and edge inference. In this work, we propose and implement PICE, an LLM serving system with semantic-level cloud-edge collaboration, enhancing inference throughput and quality through dynamic inference task scheduling, ensemble learning, and parallel edge inference. Extensive testbed experiments illustrate that our approach achieves 1.5-2× throughput enhancement and up to 43% latency reduction, while also potentially enhancing the quality compared to SOTA systems.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 33 of its 48 references that are in the map. First posted on arXiv 16 Jan 2025.

Builds on

33 of 48 references.

References · 33

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2023Qwen Technical Reportcited by 3,273
  9. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  10. 2016Concrete Problems in AI Safetycited by 1,582
  11. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  12. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
21 more
  1. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  2. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  3. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  4. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
  5. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  6. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  7. 2019A Theory of Regularized Markov Decision Processescited by 279
  8. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  9. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 154
  10. 2020SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloudcited by 109
  11. 2023Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generationcited by 103
  12. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipelinecited by 86
  13. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 71
  14. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65
  15. 2023LLMCad: Fast and Scalable On-device Large Language Model Inferencecited by 45
  16. 2019Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering Systemcited by 43
  17. 2024Harnessing the Power of Multiple Minds: Lessons Learned from LLM Routingcited by 43
  18. 2023Compresso: Structured Pruning with Collaborative Prompting Learns Compact Large Language Modelscited by 19
  19. 2024Conveyor: Efficient Tool-aware LLM Serving with Tool Partial Executioncited by 16
  20. 2024A Queueing Theoretic Perspective on Low-Latency LLM Inference with Variable Token Lengthcited by 16
  21. 2023PrivateLoRA For Efficient Privacy Preserving LLMcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025PRISM: Privacy-Aware Routing for Adaptive Cloud-Edge LLM Inference via Semantic Sketch Collaborationcited by 10
  2. 2025A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthinesscited by 5
  3. 2026PipeSD: An Efficient Cloud-Edge Collaborative Pipeline Inference Framework with Speculative Decodingcited by 4
  4. 2025Large-Small Model Collaboration for Enhancing Edge-Deployed Small Modelscited by 2
  5. 2025CoEdge-RAG: Optimizing Hierarchical Scheduling for Retrieval-Augmented LLMs in Collaborative Edge Computingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024EdgeShard: Efficient LLM Inference via Collaborative Edge Computing
  2. 2024CE-CoLLM: Efficient and Adaptive Large Language Models Through Cloud-Edge Collaboration
  3. 2025Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges
  4. 2026Pro-Router: Token-Aware Progressive Model Routing with Adaptive Edge-Cloud Collaboration for Efficient Multimodal LLM Inference
  5. 2026SPADE: Speculative Decoding for Precise and Low Cost Distributed Edge Cloud Inference
  6. 2025Taming the Titans: A Survey of Efficient LLM Inference Serving
  7. 2025Jupiter: Fast and Resource-Efficient Collaborative Inference of Generative LLMs on Edge Devices
  8. 2026End-to-End Dynamic Sparsity for Resource-Adaptive LLM Inference