mlsanityData to 14 Oct 2026

DiSCo: Device-Server Collaborative LLM-Based Text Streaming Services

arXiv 2502.11417 ↗First posted 17 Feb 2025cs.LG cs.DCcited by 2

Not on the map. Why?

Abstract (arXiv)

The rapid rise of large language models (LLMs) in text streaming services has introduced significant cost and Quality of Experience (QoE) challenges in serving millions of daily requests, especially in meeting Time-To-First-Token (TTFT) and Time-Between-Token (TBT) requirements for real-time interactions. Our real-world measurements show that both server-based and on-device deployments struggle to meet diverse QoE demands: server deployments face high costs and last-hop issues (e.g., Internet latency and dynamics), while on-device LLM inference is constrained by resources. We introduce DiSCo, a device-server cooperative scheduler designed to optimize users' QoE by adaptively routing requests and migrating response generation between endpoints while maintaining cost constraints. DiSCo employs cost-aware scheduling, leveraging the predictable speed of on-device LLM inference with the flexible capacity of server-based inference to dispatch requests on the fly, while introducing a token-level migration mechanism to ensure consistent token delivery during migration. Evaluations on real-world workloads -- including commercial services like OpenAI GPT and DeepSeek, and open-source deployments such as LLaMA3 -- show that DiSCo can improve users' QoE by reducing tail TTFT (11-52%) and mean TTFT (6-78%) across different model-device configurations, while dramatically reducing serving costs by up to 84% through its migration mechanism while maintaining comparable QoE levels.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 34 of its 69 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

34 of 69 references.

References · 34

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  4. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  5. 2018Hierarchical Neural Story Generationcited by 1,193
  6. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  7. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  8. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  9. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  10. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  11. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  12. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
22 more
  1. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  2. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 379
  3. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  4. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  5. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  6. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  7. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 214
  8. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
  9. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 103
  10. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 73
  11. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Modelscited by 71
  12. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 68
  13. 2016Weakly Supervised Cascaded Convolutional Networkscited by 47
  14. 2023LLMCad: Fast and Scalable On-device Large Language Model Inferencecited by 45
  15. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Servicescited by 45
  16. 2024SHIELD: Evaluation and Defense Strategies for Copyright Compliance in LLM Text Generationcited by 34
  17. 2024PerLLM: Personalized Inference Scheduling with Edge-Cloud Collaboration for Diverse LLM Servicescited by 23
  18. 2024WDMoE: Wireless Distributed Large Language Models with Mixture of Expertscited by 18
  19. 2024LLM for Mobile: An Initial Roadmapcited by 11
  20. 2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-locationcited by 10
  21. 2022Semi-Supervised Cascaded Clustering for Classification of Noisy Label Datacited by 2
  22. 2024Eloquent: A More Robust Transmission Scheme for LLM Token Streamingcited by 2

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Recursive Offloading for LLM Serving in Multi-tier Networkscited by 2
  2. 2025Efficient Onboard Vision-Language Inference in UAV-Enabled Low-Altitude Economy Networks via LLM-Enhanced Optimizationcited by 2