mlsanityData to 14 Oct 2026

TraCT: Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scale

arXiv 2512.18194 ↗First posted 20 Dec 2025cs.DCcited by 13

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Disaggregated LLM serving improves resource efficiency by separating the compute-intensive prefill phase from the latency-critical decode phase. However, this architecture introduces a fundamental bottleneck: key/value (KV) tensors generated during prefill must be transferred to decode workers, and existing systems rely on RDMA-based network paths for this exchange. As model sizes and context lengths increase, KV transfer dominates both time-to-first-token (TTFT) and peak throughput, and remains highly sensitive to network contention even when prefix reuse is high. This paper presents TraCT, a rack-scale LLM serving system that uses CXL shared memory as both a KV-transfer substrate and a rack-wide prefix-aware KV cache. TraCT enables GPUs to write and read KV blocks directly through CXL load/store and DMA operations, eliminating the NIC hop that constrains existing disaggregated pipelines. However, to realize this design, multiple new challenges such as synchronization, consistency, and data management on non-coherent CXL memory need to be addressed. TraCT proposes various software solutions such as the two-tier inter-node synchronization mechanism to address these challenges. We implement TraCT on the Dynamo LLM inference framework and show that, across static and synthetic workloads, TraCT reduces average TTFT by up to 9.8x, lowers P99 latency by up to 6.2x, and improves peak throughput by up to 1.6x compared to RDMA and DRAM-based caching baselines.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 8 of its 19 references that are in the map. First posted on arXiv 20 Dec 2025.

Builds on

8 of 19 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Management25 Nov
  2. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion26 May
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 5

  1. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  2. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  3. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  4. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 70
  5. 2025cMPI: Using CXL Memory Sharing for MPI One-Sided and Two-Sided Inter-Node Communicationscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 13.

13 papers.

Sort:

By area:

  1. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 4
  2. 2026Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulationcited by 3
  3. 2026HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computingcited by 2
  4. 2026SAC: Disaggregated KV Cache System for Sparse Attention LLMs with CXLcited by 2
  5. 2026AI+HW 2035: Shaping the Next Decadecited by 1
  6. 2026Stage-Aware Communication Scheduling for Disaggregated LLM Servingcited by 1
  7. 2026A CXL Memory Rack for Multi-Turn LLM Servingcited by 1
  8. 2026From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Servingcited by 0
  9. 2026Heterogeneous LLM Serving with General-Purpose Processing-Near-Memory for Retrieval-Based Sparse Attentioncited by 0
  10. 2026Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Managementcited by 0
  11. 2026Composable CXL Memory as a Kubernetes-Native Shared Memory for LLM Servingcited by 0
  12. 2026Bridging LLM Serving and CXL-SSDs with Chunk-Aware KV Cache Managementcited by 0
1 more
  1. 2026TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flashcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving
  2. 2025Scalable Processing-Near-Memory for 1M-Token LLM Inference: CXL-Enabled KV-Cache Management Beyond GPU Limits
  3. 2024KVDirect: Distributed Disaggregated LLM Inference
  4. 2025CXL-SpecKV: A Disaggregated FPGA Speculative KV-Cache for Datacenter LLM Serving
  5. 2025Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading
  6. 2026More GPUs or a Smaller Cache? Tensor Parallelism versus KV Compression for Memory-Bound LLM Serving
  7. 2026Elastic KV Cache for LLM Serving:A Working Reclamation Mechanism, and Why Chunked Prefill Already Closes the Gap
  8. 2025semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage