# Boosting LLM Serving through Spatial-Temporal GPU Resource Sharing

arXiv: https://arxiv.org/abs/2504.19516 · HTML: https://mlsanity.com/paper/2504.19516/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving.
- Builds on 21 of its 52 references that are in the map.
- First posted on arXiv 28 Apr 2025.

## Placed in

- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention](https://mlsanity.com/paper/2403.19708/) (2024)
- [LLM Inference Unveiled: Survey and Roofline Model Insights](https://mlsanity.com/paper/2402.16363/) (2024)
- [Vidur: A Large-Scale Simulation Framework For LLM Inference](https://mlsanity.com/paper/2405.05465/) (2024)
- [LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism](https://mlsanity.com/paper/2404.09526/) (2024)
- [Parrot: Efficient Serving of LLM-based Applications with Semantic Variable](https://mlsanity.com/paper/2405.19888/) (2024)
- [vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention](https://mlsanity.com/paper/2405.04437/) (2024)
- [NanoFlow: Towards Optimal Large Language Model Serving Throughput](https://mlsanity.com/paper/2408.12757/) (2024)
- [MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving](https://mlsanity.com/paper/2404.02015/) (2024)
- [A Runtime-Based Computational Performance Predictor for Deep Neural Network Training](https://mlsanity.com/paper/2102.00527/) (2021)
- [Stream-K: Work-centric Parallel Decomposition for Dense Matrix-Matrix Multiplication on the GPU](https://mlsanity.com/paper/2301.03598/) (2023)
- [Forecasting GPU Performance for Deep Learning Training and Inference](https://mlsanity.com/paper/2407.13853/) (2024)
- [Dynamic Space-Time Scheduling for GPU Inference](https://mlsanity.com/paper/1901.00041/) (2018)
- [SCOOT: SLO-Oriented Performance Tuning for LLM Inference Engines](https://mlsanity.com/paper/2408.04323/) (2024)
- [SGDRC: Software-Defined Dynamic Resource Control for Concurrent DNN Inference on NVIDIA GPUs](https://mlsanity.com/paper/2407.13996/) (2024)
- [A Framework for Fine-Grained Synchronization of Dependent GPU Kernels](https://mlsanity.com/paper/2305.13450/) (2023)

## Built on by (most cited first, 4 of 4)

- [Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving](https://mlsanity.com/paper/2507.06608/) (2025)
- [TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Serving](https://mlsanity.com/paper/2508.17219/) (2025)
- [ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Caching](https://mlsanity.com/paper/2509.16857/) (2025)
- [From Models to Operators: Rethinking Autoscaling Granularity for Large Generative Models](https://mlsanity.com/paper/2511.02248/) (2025)

## Abstract (arXiv)

Modern LLM serving systems confront inefficient GPU utilization due to the fundamental mismatch between compute-intensive prefill and memory-bound decode phases. While current practices attempt to address this by organizing these phases into hybrid batches, such solutions create an inefficient tradeoff that sacrifices either throughput or latency, leaving substantial GPU resources underutilized. We identify two key root causes: 1) the prefill phase suffers from suboptimal compute utilization due to wave quantization and attention bottlenecks. 2) hybrid batches disproportionately prioritize latency over throughput, resulting in wasted compute and memory bandwidth. To mitigate the issues, we present Bullet, a novel spatial-temporal orchestration system that eliminates these inefficiencies through precise phase coordination. Bullet enables concurrent execution of prefill and decode phases, while dynamically provisioning GPU resources using real-time performance modeling. By integrating SLO-aware scheduling and adaptive resource allocation, Bullet maximizes utilization without compromising latency targets. Experimental evaluations on real-world workloads demonstrate that Bullet delivers 1.26x average throughput gains (up to 1.55x) over state-of-the-arts, while consistently meeting latency constraints.