Cache-Resident LLM Inference in GB-Scale Last-Level Caches

arXiv 2606.25353 ↗First posted 24 Jun 2026cs.ARcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large language model (LLM) inference is increasingly dominated by data movement across the memory hierarchy. Recent 3D-stacked cache technologies have enabled GB-scale last-level caches in modern server CPUs, making it possible to keep reusable model weights on chip and exploit cache bandwidth and latency. Achieving this regime is not straightforward: deeper pipelining for weight residency increases in-flight requests and KV-cache footprint, while cache-resident operators make operator-boundary synchronization a visible bottleneck. We present a cache-resident execution model for inference on hierarchical-memory clustered systems. The model separates weight-centric operators from attention and KV-cache management into dedicated resource domains, keeping reusable weights cache-resident while scaling KV capacity independently of pipeline depth. It also relaxes synchronization from operator boundaries to true sub-operator dependencies, reducing coordination overhead in the cache-resident regime. We instantiate this model on a multi-socket CPU cluster with a weight-attention decoupled architecture, locality-aware placement, and a specialized static runtime. The prototype substantially outperforms equally provisioned llama.cpp. On deployed Llama-3.2-3B and Llama-2-7B configurations, it achieves 2.04x-11.51x speedup on time-per-output-token (TPOT). Under a validated analytical model, it further reaches up to 13.9x TPOT speedup across model sizes, context lengths, and batch sizes. These results show that commodity CPUs with GB-scale last-level caches can support efficient LLM inference when execution is organized around cache residency, decoupled state management, and dependency-aware coordination.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 14 of its 33 references that are in the map. First posted on arXiv 24 Jun 2026.

Builds on

14 of 33 references.

Closest ancestors · 1

Same direction, newest first

2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun

Other references · 13

2017Attention Is All You Needcited by 60,361
2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 98
2024T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edgecited by 55
2018Supporting Very Large Models using Automatic Dataflow Graph Partitioningcited by 43

1 more

2025WaferLLM: Large Language Model Inference at Wafer Scalecited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way