mlsanityData to 13 Oct 2026

Scalable Processing-Near-Memory for 1M-Token LLM Inference: CXL-Enabled KV-Cache Management Beyond GPU Limits

arXiv 2511.00321 ↗First posted 31 Oct 2025cs.AR cs.AIcited by 4

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The expansion of context windows in large language models (LLMs) to multi-million tokens introduces severe memory and compute bottlenecks, particularly in managing the growing Key-Value (KV) cache. While Compute Express Link (CXL) enables non-eviction frameworks that offload the full KV-cache to scalable external memory, these frameworks still suffer from costly data transfers when recalling non-resident KV tokens to limited GPU memory as context lengths increase. This work proposes scalable Processing-Near-Memory (PNM) for 1M-Token LLM Inference, a CXL-enabled KV-cache management system that coordinates memory and computation beyond GPU limits. Our design offloads token page selection to a PNM accelerator within CXL memory, eliminating costly recalls and enabling larger GPU batch sizes. We further introduce a hybrid parallelization strategy and a steady-token selection mechanism to enhance compute efficiency and scalability. Implemented atop a state-of-the-art CXL-PNM system, our solution delivers consistent performance gains for LLMs with up to 405B parameters and 1M-token contexts. Our PNM-only offloading scheme (PNM-KV) and GPU-PNM hybrid with steady-token execution (PnG-KV) achieve up to 21.9x throughput improvement, up to 60x lower energy per token, and up to 7.3x better total cost efficiency than the baseline, demonstrating that CXL-enabled multi-PNM architectures can serve as a scalable backbone for future long-context LLM inference.

Key facts. Cited by 4 papers in the map (data through 13 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 38 of its 62 references that are in the map. First posted on arXiv 31 Oct 2025.

Builds on

38 of 62 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression4 Dec
  2. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time7 Aug
  3. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  4. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  5. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
  6. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec
  7. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  8. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 30

  1. 2017Attention Is All You Needcited by 60,106
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2024The Llama 3 Herd of Modelscited by 14,486
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,376
  6. 2024Qwen2.5 Technical Reportcited by 4,840
  7. 2023Qwen Technical Reportcited by 3,310
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,197
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,137
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,386
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,323
  12. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
18 more
  1. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 361
  2. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 350
  3. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 193
  4. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  5. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 92
  6. 2024AI and Memory Wallcited by 90
  7. 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 56
  8. 2024ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Cachingcited by 48
  9. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inferencecited by 37
  10. 2024InfiniPot: Infinite Context Processing on Memory-Constrained LLMscited by 27
  11. 2025PIM Is All You Need: A CXL-Enabled GPU-Free System for Large Language Model Inferencecited by 22
  12. 2025Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantizationcited by 14
  13. 2023Demystifying CXL Memory with Genuine CXL-Ready Systems and Devicescited by 6
  14. 2025Octopus: Enhancing CXL Memory Pods via Sparse Topologycited by 3
  15. 2025Performance Characterizations and Usage Guidelines of Samsung CXL Memory Module Hybrid Prototypecited by 2
  16. 2023Failure Tolerant Training with Persistent Memory Disaggregation over CXLcited by 1
  17. 2024CXL-Interference: Analysis and Characterization in Modern Computer Systemscited by 1
  18. 2025Architectural and System Implications of CXL-enabled Tiered Memorycited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computingcited by 2
  2. 2026Bridging the Cognitive Gap: A Unified Memory Paradigm for 6G Agentic AI-RANcited by 0
  3. 2026Heterogeneous LLM Serving with General-Purpose Processing-Near-Memory for Retrieval-Based Sparse Attentioncited by 0
  4. 2026Anatomy of a Quantized Agent: VRAM Stability and Forecasting in Code-Synthesis Agentic Workloadscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching
  2. 2025Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Management
  3. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  4. 2026The KV Cache Is the New Memory Wall
  5. 2026ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference
  6. 2025Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading
  7. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
  8. 2024KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputation