mlsanityData to 14 Oct 2026

Learning how to Forget: Fine-tuning for Long-Context Sparse Attention

arXiv 2608.19920 ↗First posted 20 Aug 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

A lot of prior work addressed key-value (KV) cache selection and compression by sparse attention to enable long-context inference for transformer language models without excessive hardware budgets. We provide a new method for fine-tuning models with sparse attention. It works for any KV cache policy, runs on a moderate hardware budget (e.g., a single Nvidia A100 GPU with 40 GB RAM), and allows the model to co-adapt with the policy, often outperforming models trained with exact attention (sequence parallelism). We also provide an efficient implementation of H2O sparse attention (the leading policy in our experiments) with dedicated scaled dot product attention kernel support. KeysAndValues (https://github.com/awslabs/keys_values), a new open source library for long-context inference and fine-tuning, provides easy-to-use and performant code for all methods discussed here.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 71 of its 76 references that are in the map. First posted on arXiv 20 Aug 2026.

Builds on

71 of 76 references.

References · 71

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  12. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
38 more
  1. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  3. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  4. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  5. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  6. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  7. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  8. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  9. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  10. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  11. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  12. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  13. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  14. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  15. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  16. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  17. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 392
  18. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  19. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  20. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 355
  21. 2024xLSTM: Extended Long Short-Term Memorycited by 327
  22. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  23. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 288
  24. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  25. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  26. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 235
  27. 2024Flex Attention: A Programming Model for Generating Optimized Attention Kernelscited by 185
  28. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  29. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144
  30. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 134
  31. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 131
  32. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 126
  33. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 125
  34. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 124
  35. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 116
  36. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Headscited by 100
  37. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 76
  38. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 74

50 most cited of 71 All 71 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
  2. 2026HieraSparse: Hierarchical Semi-Structured Sparse KV Attention
  3. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving
  4. 2025HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
  5. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compression
  6. 2025NOSA: Native and Offloadable Sparse Attention
  7. 2026AB-Sparse: Sparse Attention with Adaptive Block Size for Accurate and Efficient Long-Context Inference
  8. 2026An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference