mlsanityData to 14 Oct 2026

P/D-Device: Disaggregated Large Language Model between Cloud and Devices

arXiv 2508.09035 ↗First posted 12 Aug 2025cs.DC cs.CL cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Serving disaggregated large language models has been widely adopted in industrial practice for enhanced performance. However, too many tokens generated in decoding phase, i.e., occupying the resources for a long time, essentially hamper the cloud from achieving a higher throughput. Meanwhile, due to limited on-device resources, the time to first token (TTFT), i.e., the latency of prefill phase, increases dramatically with the growth on prompt length. In order to concur with such a bottleneck on resources, i.e., long occupation in cloud and limited on-device computing capacity, we propose to separate large language model between cloud and devices. That is, the cloud helps a portion of the content for each device, only in its prefill phase. Specifically, after receiving the first token from the cloud, decoupling with its own prefill, the device responds to the user immediately for a lower TTFT. Then, the following tokens from cloud are presented via a speed controller for smoothed TPOT (the time per output token), until the device catches up with the progress. On-device prefill is then amortized using received tokens while the resource usage in cloud is controlled. Moreover, during cloud prefill, the prompt can be refined, using those intermediate data already generated, to further speed up on-device inference. We implement such a scheme P/D-Device, and confirm its superiority over other alternatives. We further propose an algorithm to decide the best settings. Real-trace experiments show that TTFT decreases at least 60%, maximum TPOT is about tens of milliseconds, and cloud throughput increases by up to 15x.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 66 of its 86 references that are in the map. First posted on arXiv 12 Aug 2025.

Builds on

66 of 86 references.

References · 66

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  10. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  11. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  12. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
38 more
  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  2. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  3. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  4. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  5. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  6. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  7. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 287
  8. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  9. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  10. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
  11. 2019Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computingcited by 152
  12. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inferencecited by 147
  13. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 124
  14. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  15. 2020SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloudcited by 112
  16. 2024On-Device Language Models: A Comprehensive Reviewcited by 104
  17. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 103
  18. 2024A Review on Edge Large Language Models: Design, Execution, and Applicationscited by 100
  19. 2023SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inferencecited by 89
  20. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partitioncited by 82
  21. 2024Fast On-device LLM Inference with NPUscited by 62
  22. 2023PanGu-Sigma: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computingcited by 56
  23. 2024Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMscited by 54
  24. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  25. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 47
  26. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devicescited by 46
  27. 2023LLMCad: Fast and Scalable On-device Large Language Model Inferencecited by 45
  28. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attentioncited by 41
  29. 2024MELTing point: Mobile Evaluation of Language Transformerscited by 36
  30. 2025GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidancecited by 32
  31. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scalecited by 27
  32. 2023Mobile Foundation Model as Firmwarecited by 25
  33. 2025Sliding Window Attention Training for Efficient Large Language Modelscited by 24
  34. 2025Serving Large Language Models on Huawei CloudMatrix384cited by 24
  35. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 23
  36. 2024SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformationcited by 19
  37. 2024FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelismcited by 19
  38. 2021DynO: Dynamic Onloading of Deep Neural Networks from Cloud to Devicecited by 18

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026WISP: Waste- and Interference-Suppressed Distributed Speculative LLM Serving at the Edge via Dynamic Drafting and SLO-Aware Batchingcited by 1
  2. 2026DySCo: Dynamic Sharding for Collaborative Edge-Cloud LLM Inference with Depth-Synchronized Batchingcited by 0