mlsanityData to 7 Oct 2026

Heterogeneous LLM Serving with General-Purpose Processing-Near-Memory for Retrieval-Based Sparse Attention

arXiv 2608.03555 ↗First posted 4 Aug 2026cs.ARcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

This paper presents a heterogeneous decode-phase serving system that relocates the KV cache out of GPU memory, motivated by the retrieval-based sparse attention that recent frontier LLMs adopt to serve million-token contexts. It partitions a decode step by operation type: GPU nodes hold the model weights and execute the projections and MoE layers, while processing-near-memory (PNM) nodes hold the KV cache and index keys and execute every operation that reads them. We first show that the assumptions behind prior PIM and PNM designs no longer hold for these operations, and derive four design requirements for such a node. From these requirements, we propose KARAT (KV-cache-resident Accelerator for Retrieval-based ATtention), a general-purpose PNM design that is the design point meeting all four. A KARAT device combines large LPDDR capacity with general-purpose compute sized for the retrieval indexer, serving an operational intensity beyond what PIM/PNM designs built for low-intensity GEMV target while accommodating diverse sparse attention algorithms that fixed-function units cannot support as they evolve. To reduce pipeline bubbles as the two device types alternate between micro-batches, we further propose opportunistic, fine-grained micro-batch scheduling (OFMS), which hides expert all-to-all behind the other micro-batch's GEMMs, and context-length-aware micro-batch rebalancing (CMR), which equalizes their token counts despite the variance in context length. Across three state-of-the-art models and real agentic traces, our proposed system improves throughput per TDP under a service-level objective by 2.09-6.13x over a GPU-only baseline and runs training-free sparse attention methods with 1.36-3.21x improvements.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 48 of its 96 references that are in the map. First posted on arXiv 4 Aug 2026.

Builds on

48 of 96 references.

Closest ancestors · 13

Same direction, newest first

  1. 2026SAC: Disaggregated KV Cache System for Sparse Attention LLMs with CXL18 Jun
  2. 2025TraCT: Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scale20 Dec
  3. 2025Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Management25 Nov
  4. 2025Scalable Processing-Near-Memory for 1M-Token LLM Inference: CXL-Enabled KV-Cache Management Beyond GPU Limits31 Oct
  5. 2025Strata: Hierarchical Context Caching for Long Context Language Model Serving26 Aug
  6. 2025Sparse Attention Remapping with Clustering for Efficient LLM Decoding on PIM9 May
  7. 2025CHIME: A Case for Efficient Long-Context Attention-FC Disaggregated Inference with DIMM-PIM24 Apr
  8. 2025A Cost-Effective Near-Storage Processing Solution for Offline Inference of Long-Context LLMs14 Feb
  9. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  10. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  11. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec
  12. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
1 more
  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 35

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  7. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  8. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  9. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  10. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  11. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  12. 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 254
23 more
  1. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  2. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  3. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  4. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
  5. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Predictioncited by 64
  6. 2024NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencingcited by 48
  7. 2023ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scalecited by 47
  8. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 43
  9. 2026SWE-chat: Coding Agent Interactions From Real Users in the Wildcited by 36
  10. 2026MiniMax Sparse Attentioncited by 34
  11. 2024Duplex: A Device for Large Language Models with Mixture of Experts, Grouped Query Attention, and Continuous Batchingcited by 33
  12. 2025InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptationcited by 31
  13. 2024LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scalecited by 30
  14. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 27
  15. 2025PIM Is All You Need: A CXL-Enabled GPU-Free System for Large Language Model Inferencecited by 22
  16. 2024IANUS: Integrated Accelerator based on NPU-PIM Unified Memory Systemcited by 11
  17. 2022TPP: Transparent Page Placement for CXL-Enabled Tiered-Memorycited by 10
  18. 2025Stratum: System-Hardware Co-Design with Tiered Monolithic 3D-Stackable DRAM for Efficient MoE Servingcited by 9
  19. 2026AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Servingcited by 9
  20. 2025Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inferencecited by 6
  21. 2026MemExplorer: Navigating the Heterogeneous Memory Design Space for Agentic Inference NPUscited by 4
  22. 2025Octopus: Enhancing CXL Memory Pods via Sparse Topologycited by 3
  23. 2024Low-overhead General-purpose Near-Data Processing in CXL Memory Expanderscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving
  2. 2025HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
  3. 2026An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference
  4. 2026AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Serving
  5. 2026HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management
  6. 2026HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computing
  7. 2026Minima-KV: Retention-Preserving KV Cache Compression with Mixed-Format Paged Attention
  8. 2025NOSA: Native and Offloadable Sparse Attention