mlsanityData to 14 Oct 2026

SPLASH: Co-Designing Sparse Attention with High-Bandwidth Flash for Efficient Long-Context Inference

arXiv 2609.23816 ↗First posted 20 Sep 2026cs.ARcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The key-value (KV) cache has become the dominant consumer of memory in large language model (LLM) serving systems as context lengths, concurrency, and request lifetimes grow. High-bandwidth memory (HBM) provides the bandwidth attention decode needs but limited capacity, while off-package memory and storage add capacity but lack the bandwidth to sustain attention decode. High-Bandwidth Flash (HBF) is a promising substrate that combines terabyte-scale capacity with near-HBM read bandwidth. Limited write endurance makes read-only model weights its natural use, but we argue that HBF paired with HBM as a hierarchy can also hold the KV cache. Unlike prior hierarchies, whose secondary tiers are bandwidth bottlenecked, the comparable bandwidths let the two act as one logical memory for the long-context KV cache. HBF capacity enables long-context serving, and sparse attention makes it efficient by limiting KV-cache reads during memory-bound decode. Since HBF reads full flash pages and aggregates bandwidth by accessing thousands of parallel flash planes, sparse attention must be co-designed with these physical properties. We present SPLASH, an algorithm and architecture co-design that virtualizes the KV cache across HBM and HBF and adapts sparse attention to HBF's page granularity and plane-level parallelism. Across models and context lengths, SPLASH improves decode throughput per GPU by 3.5x-11.4x over the evaluated baselines under a 100 ms per-token latency objective, while keeping accuracy within 4% of dense attention across long-context suites.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 53 of its 92 references that are in the map. First posted on arXiv 20 Sep 2026.

Builds on

53 of 92 references.

Closest ancestors · 15

Same direction, newest first

  1. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference18 May
  2. 2026IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs12 Apr
  3. 2025KVSwap: Disk-aware KV Cache Offloading for Long-Context On-device Inference14 Nov
  4. 2025DynaKV: Enabling Accurate and Efficient Long-Sequence LLM Decoding on Smartphones20 Oct
  5. 2025Strata: Hierarchical Context Caching for Long Context Language Model Serving26 Aug
  6. 2025Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decoding7 Jul
  7. 2025Sparse Attention Remapping with Clustering for Efficient LLM Decoding on PIM9 May
  8. 2025UniCAIM: A Unified CAM/CIM Architecture with Static-Dynamic KV Cache Pruning for Efficient Long-Context LLM Inference10 Apr
  9. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression4 Dec
  10. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  11. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  12. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
3 more
  1. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  2. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 38

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  6. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  7. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  8. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  9. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  10. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  11. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  12. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
26 more
  1. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  2. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  3. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  4. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  5. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  6. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  7. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  8. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 124
  9. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 117
  10. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 98
  11. 2024AI and Memory Wallcited by 92
  12. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systemscited by 87
  13. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 81
  14. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 68
  15. 2025ShareChat: A Dataset of Chatbot Conversations in the Wildcited by 15
  16. 2017Error Characterization, Mitigation, and Recovery in Flash Memory Based Solid-State Drivescited by 10
  17. 2026HAVEN: High-Bandwidth Flash Augmented Vector Engine for Large-Scale Approximate Nearest-Neighbor Search Accelerationcited by 6
  18. 2026MemExplorer: Navigating the Heterogeneous Memory Design Space for Agentic Inference NPUscited by 4
  19. 2026Exploring High-Bandwidth Flash for Modern LLM Inference: Opportunities and Challengescited by 4
  20. 2025On-Package Memory with Universal Chiplet Interconnect Express (UCIe): A Low Power, High Bandwidth, Low Latency and Low Cost Approachcited by 3
  21. 2026HillInfer: Efficient Long-Context LLM Inference on the Edge with Hierarchical KV Eviction using SmartSSDcited by 3
  22. 2026FlashAccel: Leveraging High-Bandwidth Flash (HBF) for High-Throughput LLM Inferencecited by 3
  23. 2026HBF Sucks? A Full-Stack Characterization of High-Bandwidth Flash for KV-Centric LLM Servingcited by 3
  24. 2026TileLens: Efficiently Using Large-Granularity Memory Systems with Transparent Two-Dimensional Memory Layoutcited by 2
  25. 2026Potential Applications of HBF in LLM Serving Systemscited by 2
  26. 2026HBFlex: A Flexible Memory System for Bridging Fine-Grained LLM States and Coarse-Grained HBF Parallel Executioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving
  2. 2026OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching
  3. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving
  4. 2026SAC: Disaggregated KV Cache System for Sparse Attention LLMs with CXL
  5. 2026FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention
  6. 2025SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving
  7. 2025Paged Attention Meets FlexAttention: Unlocking Long-Context Efficiency in Deployed Inference
  8. 2026An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference