mlsanityData to 7 Oct 2026

Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Management

arXiv 2511.20172 ↗First posted 25 Nov 2025cs.DC cs.AIcited by 11

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The rapid increase in LLM model sizes and the growing demand for long-context inference have made memory a critical bottleneck in GPU-accelerated serving systems. Although high-bandwidth memory (HBM) on GPUs offers fast access, its limited capacity necessitates reliance on host memory (CPU DRAM) to support larger working sets such as the KVCache. However, the maximum DRAM capacity is constrained by the limited number of memory channels per CPU socket. To overcome this limitation, current systems often adopt RDMA-based disaggregated memory pools, which introduce significant challenges including high access latency, complex communication protocols, and synchronization overhead. Fortunately, the emerging CXL technology introduces new opportunities in KVCache design. In this paper, we propose Beluga, a novel memory architecture that enables GPUs and CPUs to access a shared, large-scale memory pool through CXL switches. By supporting native load/store access semantics over the CXL fabric, our design delivers near-local memory latency, while reducing programming complexity and minimizing synchronization overhead. We conduct a systematic characterization of a commercial CXL switch-based memory pool and propose a set of design guidelines. Based on Beluga, we design and implement Beluga-KVCache, a system tailored for managing the large-scale KVCache in LLM inference. Beluga-KVCache achieves an 89.6% reduction in Time-To-First-Token (TTFT) and 7.35x throughput improvement in the vLLM inference engine compared to RDMA-based solutions. To the best of our knowledge, Beluga is the first system that enables GPUs to directly access large-scale memory pools through CXL switches, marking a significant step toward low-latency, shared access to vast memory resources by GPUs.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 29 of its 78 references that are in the map. First posted on arXiv 25 Nov 2025.

Builds on

29 of 78 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption25 Jul
  2. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  3. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion26 May
  4. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  5. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
  6. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  7. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  9. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 20

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  5. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  6. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  7. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  8. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  9. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 80
  10. 2023ReAcTable: Enhancing ReAct for Table Question Answeringcited by 76
  11. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 67
  12. 2025OpenSearch-SQL: Enhancing Text-to-SQL with Dynamic Few-shot and Consistency Alignmentcited by 51
8 more
  1. 2025Serving Large Language Models on Huawei CloudMatrix384cited by 22
  2. 2024λ-Tune: Harnessing Large Language Models for Automated Database System Tuningcited by 18
  3. 2022Serving Deep Learning Models with Deduplication from Relational Databasescited by 15
  4. 2022GPU-Initiated On-Demand High-Throughput Storage Access in the BaM System Architecturecited by 12
  5. 2022No Provisioned Concurrency: Fast RDMA-codesigned Remote Fork for Serverless Computingcited by 8
  6. 2024KVDirect: Distributed Disaggregated LLM Inferencecited by 8
  7. 2024DEX: Scalable Range Indexing on Disaggregated Memory [Extended Version]cited by 2
  8. 2023Transactional Indexes on (RDMA or CXL-based) Disaggregated Memory with Repairable Transactioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 10.

11 papers.

  1. 2025TraCT: Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scalecited by 13
  2. 2026SAC: Disaggregated KV Cache System for Sparse Attention LLMs with CXLcited by 2
  3. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 2
  4. 2026TENT: A Declarative Slice Spraying Engine for Performant and Resilient Data Movement in Disaggregated LLM Servingcited by 1
  5. 2026Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inferencecited by 1
  6. 2026A CXL Memory Rack for Multi-Turn LLM Servingcited by 1
  7. 2026HyperOffload: Graph-Driven Hierarchical Memory Management for Large Language Models on SuperNode Architecturescited by 0
  8. 2026From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Servingcited by 0
  9. 2026Heterogeneous LLM Serving with General-Purpose Processing-Near-Memory for Retrieval-Based Sparse Attentioncited by 0
  10. 2026Bridging LLM Serving and CXL-SSDs with Chunk-Aware KV Cache Managementcited by 0
  11. 2026TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flashcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Scalable Processing-Near-Memory for 1M-Token LLM Inference: CXL-Enabled KV-Cache Management Beyond GPU Limits
  2. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  3. 2024KunServe: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Serving
  4. 2025Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading
  5. 2025CXL-SpecKV: A Disaggregated FPGA Speculative KV-Cache for Datacenter LLM Serving
  6. 2026The KV Cache Is the New Memory Wall
  7. 2026PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding
  8. 2026Adaptive Multi-Objective Tiered Storage Configuration for KV Cache in LLM Service