mlsanityData to 14 Oct 2026

FLINT: Efficiently Leveraging High Bandwidth Flash for Capacity-Scalable LLM Inference Acceleration

arXiv 2608.25062 ↗First posted 25 Aug 2026cs.AR cs.AI cs.DCcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

LLM inference is increasingly constrained by accelerator memory capacity rather than compute throughput. This constraint is especially acute in single-accelerator and small-node inference systems, where limited on-package memory capacity restricts the size of deployable models. HBF is an emerging 3D-stacked NAND flash technology that provides multi-terabyte near-accelerator capacity, making it a promising capacity tier for storing LLM weights. However, existing HBF-based proposals face three adoption challenges: they (1) rely on coarse-grained static prefetching for LLM weights aiming to hide the microsecond-level read latency of the NAND flash device while maximizing HBF's read throughput, (2) expose NAND flash management tasks (e.g., refresh operations) to the accelerator-visible critical inference path, and (3) miss optimization opportunities to specialize and optimize the flash-management mechanisms to the workload behavior. Our goal is to design an efficient HBF substrate that integrates HBF as a memory-capacity tier alongside HBM while addressing these three challenges. To this end, we propose FLINT, a workload-driven HBF substrate for capacity-scalable LLM inference. FLINT introduces three mechanisms: (1) a hardware burst-buffer controller that dynamically coalesces and pipelines HBF reads aiming to utilize existing NAND flash buffers while sustaining high HBF bandwidth, (2) a phantom-plane refresh mechanism, which removes refresh from the critical inference path by moving refresh-related NAND flash operations outside the read foreground back via low-cost resource duplication, and (3) a read-only FTL, which replaces SSD-class support for arbitrary writes with a compact table that translates logical weight bursts to physical HBF locations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 30 of its 67 references that are in the map. First posted on arXiv 25 Aug 2026.

Builds on

30 of 67 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memory12 Dec

Other references · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2025Qwen3-VL Technical Reportcited by 2,793
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  10. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  11. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  12. 2025Kimi K2: Open Agentic Intelligencecited by 567
17 more
  1. 2022Efficiently Scaling Transformer Inferencecited by 539
  2. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  3. 2022FP8 Formats for Deep Learningcited by 196
  4. 2022Compute Trends Across Three Eras of Machine Learningcited by 163
  5. 2023Microscaling Data Formats for Deep Learningcited by 152
  6. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 103
  7. 2024AI and Memory Wallcited by 92
  8. 2023Ramulator 2.0: A Modern, Modular, and Extensible DRAM Simulatorcited by 30
  9. 2017Error Characterization, Mitigation, and Recovery in Flash Memory Based Solid-State Drivescited by 10
  10. 2025PRESERVE: Prefetching Model Weights and KV-Cache in Distributed LLM Servingcited by 9
  11. 2026HAVEN: High-Bandwidth Flash Augmented Vector Engine for Large-Scale Approximate Nearest-Neighbor Search Accelerationcited by 6
  12. 2024MegIS: High-Performance, Energy-Efficient, and Low-Cost Metagenomic Analysis with In-Storage Processingcited by 5
  13. 2018Read Disturb Errors in MLC NAND Flash Memorycited by 4
  14. 2025REIS: A High-Performance and Energy-Efficient Retrieval System with In-Storage Processingcited by 4
  15. 2026MemExplorer: Navigating the Heterogeneous Memory Design Space for Agentic Inference NPUscited by 4
  16. 2026Exploring High-Bandwidth Flash for Modern LLM Inference: Opportunities and Challengescited by 4
  17. 2025MARS: Processing-In-Memory Acceleration of Raw Signal Genome Analysis Inside the Storage Subsystemcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026FlashAccel: Leveraging High-Bandwidth Flash (HBF) for High-Throughput LLM Inference
  2. 2026Beyond Capacity: Scalable MoE LLM Inference via High-Bandwidth Flash with Direct GPU and HBM Paths
  3. 2026Potential Applications of HBF in LLM Serving Systems
  4. 2026Characterizing High Bandwidth Flash for LLM Serving
  5. 2025MIST: A Co-Design Framework for Heterogeneous, Multi-Stage LLM Inference
  6. 2026HBFSim: Fast and Faithful Simulation of High-Bandwidth Flash Under Real GPU Execution ↗
  7. 2025Architecting Long-Context LLM Acceleration with Packing-Prefetch Scheduler and Ultra-Large Capacity On-Chip Memories
  8. 2025LoopLynx: A Scalable Dataflow Architecture for Efficient LLM Inference