mlsanityData to 13 Oct 2026

CXL-SpecKV: A Disaggregated FPGA Speculative KV-Cache for Datacenter LLM Serving

arXiv 2512.11920 ↗First posted 11 Dec 2025cs.AIcited by 8

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large Language Models (LLMs) have revolutionized natural language processing tasks, but their deployment in datacenter environments faces significant challenges due to the massive memory requirements of key-value (KV) caches. During the autoregressive decoding process, KV caches consume substantial GPU memory, limiting batch sizes and overall system throughput. To address these challenges, we propose CXL-SpecKV, a novel disaggregated KV-cache architecture that leverages Compute Express Link (CXL) interconnects and FPGA accelerators to enable efficient speculative execution and memory disaggregation. Our approach introduces three key innovations: (i) a CXL-based memory disaggregation framework that offloads KV-caches to remote FPGA memory with low latency, (ii) a speculative KV-cache prefetching mechanism that predicts and preloads future tokens' cache entries, and (iii) an FPGA-accelerated KV-cache compression and decompression engine that reduces memory bandwidth requirements by up to 4×. When evaluated on state-of-the-art LLM models, CXL-SpecKV achieves up to 3.2× higher throughput compared to GPU-only baselines, while reducing memory costs by 2.8× and maintaining accuracy. Our system demonstrates that intelligent memory disaggregation combined with speculative execution can effectively address the memory wall challenge in large-scale LLM serving. Our code implementation has been open-sourced at https://github.com/FastLM/CXL-SpecKV.

Key facts. Cited by 8 papers in the map (data through 13 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 20 of its 48 references that are in the map. First posted on arXiv 11 Dec 2025.

Builds on

20 of 48 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025TinyServe: Query-Aware Cache Selection for Efficient LLM Serving28 Aug
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  3. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2024The Llama 3 Herd of Modelscited by 14,486
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,133
  4. 2023Qwen Technical Reportcited by 3,310
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,137
  6. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,625
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,365
  8. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,115
  9. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 829
  10. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 810
  11. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 609
  12. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 572
5 more
  1. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 570
  2. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 406
  3. 2018Learning Memory Access Patternscited by 28
  4. 2025CSV-Decode: Certifiable Sub-Vocabulary Decoding for Efficient Large Language Model Inferencecited by 7
  5. 2025Systolic Sparse Tensor Slices: FPGA Building Blocks for Sparse and Dense AI Accelerationcited by 4

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026AdaCorrection: Adaptive Offset Cache Correction for Accurate Diffusion Transformerscited by 3
  2. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 3
  3. 2026MedRAGChecker: Claim-Level Verification for Biomedical Retrieval-Augmented Generationcited by 1
  4. 2026Accelerating Frequency Domain Diffusion Models with Error-Feedback Event-Driven Cachingcited by 1
  5. 2026Predictive Multi-Tier Memory Management for KV Cache in Large-Scale GPU Inferencecited by 1
  6. 2026Thoughts-as-Planning: Latent World Models for Chain-of-Thoughts Optimization via Reinforcement Planningcited by 1
  7. 2026From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Servingcited by 0
  8. 2026MeshKV: A Network-on-Chip KV Cache Fabric for Scalable Transformer Decoding Acceleratorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025TraCT: Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scale
  2. 2025Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Management
  3. 2025Scalable Processing-Near-Memory for 1M-Token LLM Inference: CXL-Enabled KV-Cache Management Beyond GPU Limits
  4. 2026A CXL Memory Rack for Multi-Turn LLM Serving
  5. 2025SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs
  6. 2025Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implications
  7. 2024KVDirect: Distributed Disaggregated LLM Inference
  8. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving