mlsanityData to 14 Oct 2026

Hot-Cold Tiering of HBM and High Bandwidth Flash for Agentic LLM Serving

arXiv 2609.25782 ↗First posted 22 Sep 2026cs.ARcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large language model (LLM) serving is increasingly agentic, with multi-turn sessions that idle between actions yet must retain their full context. Limited GPU memory capacity forces inactive KV states to be evicted, so resuming a session incurs either costly recomputation or slow interconnect transfers. To address this, high bandwidth flash (HBF)-an on-package 3D-NAND memory offering orders-of-magnitude greater capacity than high bandwidth memory (HBM) at comparable read bandwidth-has emerged as a strong candidate. However, its high read energy and limited write endurance make it impractical to serve all KV traffic. Fortunately, our analysis shows that agentic KV states exhibit distinct access patterns: a small hot set is read for every decoding step, while a large cold pool is read only when a paused session resumes. Exploiting this, we place the hot set in HBM and the cold pool in HBF, forming a hot-cold KV hierarchy within the GPU memory tier. On agentic workloads with Qwen3-Coder-30B-A3B, our design delivers 14 ms time-between-tokens (TBT) and adds only ≈0.1 ms of resume latency on top of prefill, while hosting 24× more concurrent sessions per GPU. By confining HBM to the hot set, our design also cuts read power by 7.6 kW per 8-GPU node relative to serving all KV from flash-establishing HBF as a cold-tier complement to HBM rather than its replacement.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 15 of its 25 references that are in the map. First posted on arXiv 22 Sep 2026.

Builds on

15 of 25 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026FlashAccel: Leveraging High-Bandwidth Flash (HBF) for High-Throughput LLM Inference11 Jul
  2. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar

Other references · 13

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023Stateful Large Language Model Serving with Pensievecited by 61
  4. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 44
  5. 2025The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspectivecited by 39
  6. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applicationscited by 16
  7. 2026CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Controlcited by 14
  8. 2026Challenges and Research Directions for Large Language Model Inference Hardwarecited by 11
  9. 2026HAVEN: High-Bandwidth Flash Augmented Vector Engine for Large-Scale Approximate Nearest-Neighbor Search Accelerationcited by 6
  10. 2026Exploring High-Bandwidth Flash for Modern LLM Inference: Opportunities and Challengescited by 4
  11. 2026Characterizing CPU-Induced Slowdowns in Multi-GPU LLM Inferencecited by 3
  12. 2026TileLens: Efficiently Using Large-Granularity Memory Systems with Transparent Two-Dimensional Memory Layoutcited by 2
1 more
  1. 2026RPU -- A Reasoning Processing Unitcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Characterizing High Bandwidth Flash for LLM Serving
  2. 2026HBFlex: A Flexible Memory System for Bridging Fine-Grained LLM States and Coarse-Grained HBF Parallel Execution
  3. 2026Potential Applications of HBF in LLM Serving Systems
  4. 2025KVNAND: Efficient On-Device Large Language Model Inference Using DRAM-Free In-Flash Computing
  5. 2026HBF Sucks? A Full-Stack Characterization of High-Bandwidth Flash for KV-Centric LLM Serving
  6. 2026TokenStack: A Heterogeneous HBM-PIM Architecture and Runtime for Efficient LLM Inference
  7. 2026SPLASH: Co-Designing Sparse Attention with High-Bandwidth Flash for Efficient Long-Context Inference
  8. 2026FLINT: Efficiently Leveraging High Bandwidth Flash for Capacity-Scalable LLM Inference Acceleration