mlsanityData to 14 Oct 2026

Characterizing High Bandwidth Flash for LLM Serving

arXiv 2609.39131 ↗First posted 30 Sep 2026cs.LG cs.AR cs.DCcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large language model (LLM) serving requires substantial memory to store model weights and KV caches. As models grow larger and contexts become longer, memory capacity and bandwidth increasingly become bottlenecks for serving performance. Agentic workloads compound this pressure through repeated interactions over growing contexts, making it increasingly important to retain KV state for reuse. High-bandwidth flash (HBF) offers a way to expand accelerator memory capacity for large language model (LLM) serving, but its access costs and limited write endurance complicate its use. We evaluate HBF for high-throughput agentic serving across system design and scheduling choices to understand when additional capacity improves serving performance and energy efficiency. We introduce an HBM-HBF-host hierarchical storage system and buffered cache-aware scheduling, and use trace-driven simulations to analyze their effects on performance, energy consumption, and HBF write lifetime. Across the evaluated workloads, the fastest HBF-augmented systems reduce completion time by 36.1-87.7% relative to HBM-only systems. Modeled energy savings reach 59.1%, with benefits depending on the workload and weight placement. Buffered cache-aware scheduling extends estimated HBF write lifetime from 1.21 to 14.82 years in the evaluated configuration. These results demonstrate the importance of coordinating data placement and scheduling to improve serving efficiency while sustaining a practical HBF write lifetime.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 21 of its 40 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

21 of 40 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026LLM Inference in a Flash!14 Sep
  2. 2026LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructure26 Feb
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  4. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memory12 Dec

Other references · 17

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  5. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  6. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223
  7. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
  8. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 126
  9. 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programscited by 58
  10. 2023ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scalecited by 47
  11. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 44
  12. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 31
5 more
  1. 2026ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference Systemcited by 17
  2. 2026Challenges and Research Directions for Large Language Model Inference Hardwarecited by 11
  3. 2026HAVEN: High-Bandwidth Flash Augmented Vector Engine for Large-Scale Approximate Nearest-Neighbor Search Accelerationcited by 6
  4. 2026MemExplorer: Navigating the Heterogeneous Memory Design Space for Agentic Inference NPUscited by 4
  5. 2026HBF Sucks? A Full-Stack Characterization of High-Bandwidth Flash for KV-Centric LLM Servingcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Potential Applications of HBF in LLM Serving Systems
  2. 2026Hot-Cold Tiering of HBM and High Bandwidth Flash for Agentic LLM Serving
  3. 2026HBFlex: A Flexible Memory System for Bridging Fine-Grained LLM States and Coarse-Grained HBF Parallel Execution
  4. 2025FastCache: Optimizing Multimodal LLM Serving through Lightweight KV-Cache Compression Framework
  5. 2026FlashAccel: Leveraging High-Bandwidth Flash (HBF) for High-Throughput LLM Inference
  6. 2026Understanding Efficiency: Quantization, Batching, and Serving Strategies in LLM Energy Use
  7. 2026FLINT: Efficiently Leveraging High Bandwidth Flash for Capacity-Scalable LLM Inference Acceleration
  8. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving