mlsanityData to 7 Oct 2026

Splitwise: Efficient generative LLM inference using phase splitting

arXiv 2311.18677 ↗First posted 30 Nov 2023cs.AR cs.DCcited by 536

Not on the map. Why?

Cited by 280 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent innovations in generative large language models (LLMs) have made their applications and use-cases ubiquitous. This has led to large-scale deployments of these models, using complex, expensive, and power-hungry AI accelerators, most commonly GPUs. These developments make LLM inference efficiency an important challenge. Based on our extensive characterization, we find that there are two main phases during an LLM inference request: a compute-intensive prompt computation, and a memory-intensive token generation, each with distinct latency, throughput, memory, and power characteristics. Despite state-of-the-art batching and scheduling, the token generation phase underutilizes compute resources. Specifically, unlike compute-intensive prompt computation phases, token generation phases do not require the compute capability of the latest GPUs, and can be run with lower power and cost. With Splitwise, we propose splitting the two phases of a LLM inference request on to separate machines. This allows us to use hardware that is well-suited for each phase, and provision resources independently per phase. However, splitting an inference request across machines requires state transfer from the machine running prompt computation over to the machine generating tokens. We implement and optimize this state transfer using the fast back-plane interconnects available in today's GPU clusters. We use the Splitwise technique to design LLM inference clusters using the same or different types of machines for the prompt computation and token generation phases. Our clusters are optimized for three key objectives: throughput, cost, and power. In particular, we show that we can achieve 1.4x higher throughput at 20% lower cost than current designs. Alternatively, we can achieve 2.35x more throughput with the same cost and power budgets.

Key facts. Cited by 536 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Nov 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 84; 2025: 183; 2026: 267.

536 papers.

Sort:

By area:

  1. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  2. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  3. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  4. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  5. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  6. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  7. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  8. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
  9. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 115
  10. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  11. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 110
  12. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
38 more
  1. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  2. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
  3. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 96
  4. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
  5. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 85
  6. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  7. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systemscited by 84
  8. 2024Efficient LLM Scheduling by Learning to Rankcited by 83
  9. 2025How Hungry is AI? Benchmarking Energy, Water, and Carbon Footprint of LLM Inferencecited by 73
  10. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 68
  11. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compressioncited by 65
  12. 2024Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inferencecited by 59
  13. 2024Fast On-device LLM Inference with NPUscited by 58
  14. 2024Efficiently Scaling LLM Reasoning with Certaindexcited by 56
  15. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generationcited by 56
  16. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 54
  17. 2025Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decodingcited by 52
  18. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  19. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  20. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 46
  21. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  22. 2024EPIC: Efficient Position-Independent Caching for Serving Large Language Modelscited by 45
  23. 2024Optimizing RLHF Training for Large Language Models with Stage Fusioncited by 41
  24. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
  25. 2024SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attentioncited by 39
  26. 2025The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspectivecited by 38
  27. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inferencecited by 37
  28. 2024A Declarative System for Optimizing AI Workloadscited by 35
  29. 2024Duplex: A Device for Large Language Models with Mixture of Experts, Grouped Query Attention, and Continuous Batchingcited by 33
  30. 2024InferCept: Efficient Intercept Support for Augmented Large Language Model Inferencecited by 32
  31. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Expertscited by 31
  32. 2025MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelismcited by 31
  33. 2025The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimizationcited by 31
  34. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 30
  35. 2024LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scalecited by 30
  36. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Productioncited by 29
  37. 2025TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platformscited by 28
  38. 2024Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systemscited by 27

50 most cited of 536 All 536 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottlenecks
  2. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective
  3. 2026Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
  4. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline
  5. 2025DeServe: Towards Affordable Offline LLM Inference via Decentralization
  6. 2024LLM Inference Unveiled: Survey and Roofline Model Insights
  7. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
  8. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism