mlsanityData to 14 Oct 2026

An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference

arXiv 2605.07719 ↗First posted 8 May 2026cs.LG cs.AI cs.PFcited by 3

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Long-context inference increasingly operates over CPU-resident KV caches, either because decoding-time KV states exceed GPU memory capacity or because disaggregated prefill-decode systems place KV data in host memory. Although block-sparse attention reduces attention cost in this setting, sparsity alone is insufficient for end-to-end efficiency. GPU-only designs remain constrained by PCIe bandwidth and metadata memory overhead, while CPU-GPU hybrid designs still suffer from substantial GPU idle time and bottlenecks in CPU-side top-k selection and sparse attention computation. Fluxion is built on three key insights: output-aware KV budgeting, head-specific and granularity-aware sparse configuration, and cross-device coordinated execution for sparse attention over CPU-resident KV caches. Guided by these insights, Fluxion combines a lightweight head-property predictor, a granularity-budget selector, and a priority-based scheduler to jointly optimize budget allocation, sparse configuration, and CPU-GPU execution overlap. This co-design enables hybrid sparse attention to achieve both accuracy and system efficiency in long-context inference. Across 2 models, 3 benchmarks, and 40 tasks, Fluxion preserves quality well -- the worst average degradation is only -0.26 relative to FULL, while delivering 1.5×-3.7× speedup over the strongest fixed sparse hybrid baseline, whose KV budget is only 0.05.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 40 of its 51 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

40 of 51 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025FlexiCache: Leveraging Temporal Stability of Attention Heads for Efficient KV Cache Management2 Nov
  2. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference8 Oct
  3. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving1 Mar
  4. 2024DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction4 Dec
  5. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  6. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  7. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  8. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  9. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  10. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec

Other references · 30

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  8. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  9. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  10. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  11. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  12. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
18 more
  1. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  2. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 198
  3. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  4. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 137
  5. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 124
  6. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 120
  7. 2025A Comprehensive Survey on Long Context Language Modelingcited by 117
  8. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
  9. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  10. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 89
  11. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  12. 2023FlashDecoding++: Faster Large Language Model Inference on GPUscited by 71
  13. 2021FBGEMM: Enabling High-Performance Low-Precision Deep Learning Inferencecited by 41
  14. 2024SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attentioncited by 39
  15. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inferencecited by 37
  16. 2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 30
  17. 2024RuleR: Improving LLM Controllability by Rule-based Data Recyclingcited by 11
  18. 2025HGCA: Hybrid GPU-CPU Attention for Long Context LLM Inferencecited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Move the Query, Not the Cache: Characterizing Cross-Instance Latent Attention Redistribution Across GPU Fabricscited by 1
  2. 2026Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inferencecited by 0
  3. 2026MOIRA: Mass-Oriented Indexing with Ragged Attention for Long-Context Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving
  2. 2025NOSA: Native and Offloadable Sparse Attention
  3. 2026HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management
  4. 2026Predictive Multi-Tier Memory Management for KV Cache in Large-Scale GPU Inference
  5. 2026Stochastic Sparse Attention for Memory-Bound Inference
  6. 2026Learning how to Forget: Fine-tuning for Long-Context Sparse Attention
  7. 2026SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference
  8. 2025HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs