mlsanityData to 14 Oct 2026

NOSA: Native and Offloadable Sparse Attention

arXiv 2510.13602 ↗First posted 15 Oct 2025cs.CL cs.AI cs.LGcited by 4

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Decoding throughput improvements from larger inference batches are limited by GPU memory, which is largely consumed by the key-value (KV) cache. Prior training-free KV cache offloading alleviates this by keeping redundant context on the CPU and fetching only a sparse subset for attention, but it often degrades long-generation quality due to training-inference mismatch on sparse patterns. Meanwhile, trainable sparse attention is incompatible with efficient offloading, as unconstrained KV accesses may force large CPU-to-GPU transfers and erase throughput gains. To this end, we propose NOSA, a trainable sparse attention mechanism natively designed for KV cache offloading. NOSA explicitly constrains the volume of CPU-GPU KV transfers, thereby achieving low communication overhead and high decoding throughput. We further build NOSI, a KV cache offloading inference system that fully unlocks NOSA's efficiency. Empirical results on 1,3,8B LLMs demonstrate that NOSA outperforms KV cache offloading baselines on general, long-input, and long-generation tasks, while boosting decoding throughput by up to 5.04x, 1.92x, and 1.83x over FullAttn, InfLLMv2, and ShadowKV, respectively. We release our code at https://github.com/thunlp/NOSA.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 85 of its 101 references that are in the map. First posted on arXiv 15 Oct 2025.

Builds on

85 of 101 references.

Closest ancestors · 21

Same direction, newest first

  1. 2025LouisKV: Efficient KV Cache Retrieval for Long Input-Output Sequences13 Oct
  2. 2025SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving29 Sep
  3. 2025Sparse Attention across Multiple-context KV Cache6 Aug
  4. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention20 Feb
  5. 2025Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs20 Feb
  6. 2025Task-KV: Task-aware KV Cache Optimization via Semantic Differentiation of Attention Heads25 Jan
  7. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving2 Jan
  8. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression4 Dec
  9. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  10. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  11. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration3 Oct
  12. 2024Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devices2 Oct
9 more
  1. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compression10 Aug
  2. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  3. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models23 May
  4. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Models17 May
  5. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  6. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  8. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  9. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 64

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2024GPT-4o System Cardcited by 4,588
  9. 2023Let's Verify Step by Stepcited by 3,789
  10. 2021Program Synthesis with Large Language Modelscited by 3,637
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  12. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
38 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  2. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  5. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  7. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  8. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  9. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  10. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  11. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  12. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  13. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  14. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  15. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  16. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  17. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  18. 2023Tool Learning with Foundation Modelscited by 302
  19. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 250
  20. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  21. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 205
  22. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 198
  23. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  24. 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 186
  25. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  26. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 137
  27. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 125
  28. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 124
  29. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 124
  30. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 114
  31. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  32. 2025VSA: Faster Video Diffusion with Trainable Sparse Attentioncited by 104
  33. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 89
  34. 2025SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Trainingcited by 66
  35. 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 57
  36. 2024QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overheadcited by 52
  37. 2025MiniCPM4: Ultra-Efficient LLMs on End Devicescited by 52
  38. 2024Post-Training Sparse Attention with Double Sparsitycited by 45

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inferencecited by 7
  2. 2026SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inferencecited by 5
  3. 2026DashAttention: Differentiable and Adaptive Sparse Hierarchical Attentioncited by 4
  4. 2026Vortex: Efficient and Programmable Sparse Attention Serving for AI Agentscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference
  2. 2026Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving
  3. 2026Stochastic Sparse Attention for Memory-Bound Inference
  4. 2026SAC: Disaggregated KV Cache System for Sparse Attention LLMs with CXL
  5. 2025HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
  6. 2026Learning how to Forget: Fine-tuning for Long-Context Sparse Attention
  7. 2026HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management
  8. 2026ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference