# Characterizing CPU-Induced Slowdowns in Multi-GPU LLM Inference

arXiv: https://arxiv.org/abs/2603.22774 · HTML: https://mlsanity.com/paper/2603.22774/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Mar 2026.

## Built on by (most cited first, 3 of 3)

- [Taking Tokenization off the Host for Agentic LLM Serving](https://mlsanity.com/paper/2610.13325/) (2026)
- [Hot-Cold Tiering of HBM and High Bandwidth Flash for Agentic LLM Serving](https://mlsanity.com/paper/2609.25782/) (2026)
- [Delta Tokenizer: Reference-Identical Tokenization of Session Deltas and Segment-Parallelism for Long Prompts on Multicore CPUs](https://mlsanity.com/paper/2610.19801/) (2026)

## Abstract (arXiv)

Large-scale machine learning workloads increasingly rely on multi-GPU systems, yet their performance is often limited by an overlooked component: the CPU. Through a detailed study of modern large language model (LLM) serving workloads, we find that multi-GPU performance often degrades not because GPUs are saturated, but because CPUs fail to keep them busy. Under limited CPU allocations, systems exhibit symptoms such as delayed kernel launch, stalled communication, and increased tokenization latency, leading to severe GPU underutilization even when ample GPU resources are available. The problem becomes more severe in agentic LLM serving, where long accumulated contexts increase CPU-side tokenization work while high prefix-cache reuse across multi-turn interactions reduces GPU-side prefill work. These bottlenecks persist even in serving stacks that employ process-level separation and modern GPU-side optimizations such as CUDA Graphs. Since CPU cores cost orders of magnitude less than GPUs, provisioning additional cores is a highly cost-effective mitigation. Under moderate serving load, we observe that CPU-starved configurations frequently time out, while providing adequate CPU resources restores responsiveness and reduces time-to-first-token (TTFT) latency by 1.47-7.11x across configurations, all without requiring additional GPUs.