# Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Management

arXiv: https://arxiv.org/abs/2511.20172 · HTML: https://mlsanity.com/paper/2511.20172/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving.
- Builds on 29 of its 78 references that are in the map.
- First posted on arXiv 25 Nov 2025.

## Placed in

- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 29 of 29)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://mlsanity.com/paper/2401.18079/) (2024)
- [Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs](https://mlsanity.com/paper/2310.01801/) (2023)
- [InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management](https://mlsanity.com/paper/2406.19707/) (2024)
- [CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion](https://mlsanity.com/paper/2405.16444/) (2024)
- [CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving](https://mlsanity.com/paper/2310.07240/) (2023)
- [Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference](https://mlsanity.com/paper/2403.09054/) (2024)
- [Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption](https://mlsanity.com/paper/2407.18003/) (2024)
- [LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K](https://mlsanity.com/paper/2402.05136/) (2024)
- [ReAcTable: Enhancing ReAct for Table Question Answering](https://mlsanity.com/paper/2310.00815/) (2023)
- [MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool](https://mlsanity.com/paper/2406.17565/) (2024)
- [OpenSearch-SQL: Enhancing Text-to-SQL with Dynamic Few-shot and Consistency Alignment](https://mlsanity.com/paper/2502.14913/) (2025)
- [Serving Large Language Models on Huawei CloudMatrix384](https://mlsanity.com/paper/2506.12708/) (2025)
- [λ-Tune: Harnessing Large Language Models for Automated Database System Tuning](https://mlsanity.com/paper/2411.03500/) (2024)
- [Serving Deep Learning Models with Deduplication from Relational Databases](https://mlsanity.com/paper/2201.10442/) (2022)
- [GPU-Initiated On-Demand High-Throughput Storage Access in the BaM System Architecture](https://mlsanity.com/paper/2203.04910/) (2022)
- [No Provisioned Concurrency: Fast RDMA-codesigned Remote Fork for Serverless Computing](https://mlsanity.com/paper/2203.10225/) (2022)
- [KVDirect: Distributed Disaggregated LLM Inference](https://mlsanity.com/paper/2501.14743/) (2024)
- [DEX: Scalable Range Indexing on Disaggregated Memory [Extended Version]](https://mlsanity.com/paper/2405.14502/) (2024)
- [Transactional Indexes on (RDMA or CXL-based) Disaggregated Memory with Repairable Transaction](https://mlsanity.com/paper/2308.02501/) (2023)

## Built on by (most cited first, 11 of 11)

- [TraCT: Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scale](https://mlsanity.com/paper/2512.18194/) (2025)
- [Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization](https://mlsanity.com/paper/2607.08057/) (2026)
- [SAC: Disaggregated KV Cache System for Sparse Attention LLMs with CXL](https://mlsanity.com/paper/2606.19746/) (2026)
- [TENT: A Declarative Slice Spraying Engine for Performant and Resilient Data Movement in Disaggregated LLM Serving](https://mlsanity.com/paper/2604.00368/) (2026)
- [Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference](https://mlsanity.com/paper/2607.01831/) (2026)
- [A CXL Memory Rack for Multi-Turn LLM Serving](https://mlsanity.com/paper/2607.18141/) (2026)
- [HyperOffload: Graph-Driven Hierarchical Memory Management for Large Language Models on SuperNode Architectures](https://mlsanity.com/paper/2602.00748/) (2026)
- [From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving](https://mlsanity.com/paper/2607.02574/) (2026)
- [Heterogeneous LLM Serving with General-Purpose Processing-Near-Memory for Retrieval-Based Sparse Attention](https://mlsanity.com/paper/2608.03555/) (2026)
- [Bridging LLM Serving and CXL-SSDs with Chunk-Aware KV Cache Management](https://mlsanity.com/paper/2609.26828/) (2026)
- [TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flash](https://mlsanity.com/paper/2609.35065/) (2026)

## Abstract (arXiv)

The rapid increase in LLM model sizes and the growing demand for long-context inference have made memory a critical bottleneck in GPU-accelerated serving systems. Although high-bandwidth memory (HBM) on GPUs offers fast access, its limited capacity necessitates reliance on host memory (CPU DRAM) to support larger working sets such as the KVCache. However, the maximum DRAM capacity is constrained by the limited number of memory channels per CPU socket. To overcome this limitation, current systems often adopt RDMA-based disaggregated memory pools, which introduce significant challenges including high access latency, complex communication protocols, and synchronization overhead. Fortunately, the emerging CXL technology introduces new opportunities in KVCache design. In this paper, we propose Beluga, a novel memory architecture that enables GPUs and CPUs to access a shared, large-scale memory pool through CXL switches. By supporting native load/store access semantics over the CXL fabric, our design delivers near-local memory latency, while reducing programming complexity and minimizing synchronization overhead. We conduct a systematic characterization of a commercial CXL switch-based memory pool and propose a set of design guidelines. Based on Beluga, we design and implement Beluga-KVCache, a system tailored for managing the large-scale KVCache in LLM inference. Beluga-KVCache achieves an 89.6% reduction in Time-To-First-Token (TTFT) and 7.35x throughput improvement in the vLLM inference engine compared to RDMA-based solutions. To the best of our knowledge, Beluga is the first system that enables GPUs to directly access large-scale memory pools through CXL switches, marking a significant step toward low-latency, shared access to vast memory resources by GPUs.