mlsanityData to 7 Oct 2026

MiniMax Sparse Attention

arXiv 2606.13392 ↗First posted 11 Jun 2026cs.AIcited by 34

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Ultra-long-context capability is becoming indispensable for frontier LLMs: agentic workflows, repository-scale code reasoning, and persistent memory all require the model to jointly attend over hundreds of thousands to millions of tokens, yet the quadratic cost of softmax attention makes this untenable at deployment scale. We introduce MiniMax Sparse Attention (MSA), a blockwise sparse attention built upon Grouped Query Attention (GQA). A lightweight Index Branch scores key-value blocks and independently selects a Top-k subset for each GQA group, enabling group-specific sparse retrieval while maintaining efficient block-level execution; the Main Branch then performs exact block-sparse attention over only the selected blocks. Designed around a principle of simplicity and scalability, MSA is deliberately streamlined, making it straightforward to deploy efficiently across a broad range of GPUs. To translate sparsity into practical speedups, we co-design MSA with a GPU execution path that uses exp-free Top-k selection and KV-outer sparse attention to improve tensor-core utilization under block-granular access. On a 109B-parameter model with native multimodal training, MSA performs on par with GQA while reducing per-token attention compute by 28.4x at 1M context. Paired with our co-designed kernel, MSA achieves 14.2x prefill and 7.6x decoding wall-clock speedups on H800. Our inference kernel is available at: https://github.com/MiniMax-AI/MSA. A production-grade natively multimodal model powered by MSA has been publicly released at: https://huggingface.co/MiniMaxAI/MiniMax-M3.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 52 of its 63 references that are in the map. First posted on arXiv 11 Jun 2026.

Builds on

52 of 63 references.

References · 52

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  9. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
38 more
  1. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  2. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  3. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  4. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  5. 2020Rethinking Attention with Performerscited by 1,584
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  7. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  8. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  9. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  11. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  12. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  13. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  14. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  15. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  16. 2016A Diagram Is Worth A Dozen Imagescited by 793
  17. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  18. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
  19. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
  20. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  21. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  22. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  23. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  24. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  25. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  26. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 306
  27. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
  28. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 229
  29. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  30. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  31. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  32. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  33. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 131
  34. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  35. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 98
  36. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Modelscited by 96
  37. 2025Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Modelscited by 65
  38. 2025TileLang: A Composable Tiled Programming Model for AI Systemscited by 42

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 34.

34 papers.

Sort:

By area:

  1. 2026Evo-Bench: Can Language Models Improve Agent Harness?cited by 8
  2. 2026StructAgent: Harness Long-horizon Digital Agents with Unified Causal Structurecited by 4
  3. 2026Predict, Don't Iterate: Efficient Adaptive-Length Infilling for Diffusion Language Modelscited by 3
  4. 2026LiteTopK: Exploiting the Curse of Dimensionality for a Fused Indexer-TopK Kernel in Long-Context Sparse Attentioncited by 1
  5. 2026OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Groundingcited by 1
  6. 2026Language Models Can Control Their Own Attentioncited by 1
  7. 2026Attention is Just Another Name for Coupling? A Fast-Slow ODE Perspective on Hierarchical Pretrainingcited by 0
  8. 2026Interpretable Column Annotation with LLM-Symbolized Decision Process Materializationcited by 0
  9. 2026Voice Memory for Agentic Speech Recognitioncited by 0
  10. 2026Heterogeneous LLM Serving with General-Purpose Processing-Near-Memory for Retrieval-Based Sparse Attentioncited by 0
  11. 2026ELBench: A Multi-Dimensional Benchmark for Education-Facing Large Language Modelscited by 0
  12. 2026HarnessRisk: A Lifecycle-Oriented Benchmark for Agent Harness Safetycited by 0
22 more
  1. 2026SemaPLC: A Project-Grounded, Verification-Gated Agent Harness for PLC Code Generationcited by 0
  2. 2026ClawSentry: A Progressive Multi-Tier Security Monitor for Safeguarding Autonomous LLM Agentscited by 0
  3. 2026LongWoF-Bench: Evaluating EvoMap Genes for Verifiable Long-Workflow Taskscited by 0
  4. 2026EMRB: A Multi-Level Benchmark for Evaluating LLM Reasoning over Raw Electromagnetic Signalscited by 0
  5. 2026Super Library Agent: Joint Generation and Maintenance of Multiple Applications Beyond the Single Codebasecited by 0
  6. 2026LLMs Interpret, Embeddings Organize, Graphs Emerge: Agent-Driven Compilation of Scientific Knowledgecited by 0
  7. 2026Opinion Leader Dynamics: How Sparse Attention Shapes Token Clusteringcited by 0
  8. 2026TimeLitmus: A Diagnostic Benchmark for Cross-Modal Understanding and Explanation Faithfulness in Event-Conditioned Time-Series Predictioncited by 0
  9. 2026FinFIRST: Benchmarking Search Agents for Financial Information Retrieval, Sourcing and Traceabilitycited by 0
  10. 2026Video-HopChain: Multi-Hop Questions and Confidence-Gated Exploration for Video Reasoning Modelscited by 0
  11. 2026Learn How to Act from Your Own Interactions: On-Policy Self-Distillation for GUI Agentscited by 0
  12. 2026CCRV-Bench: Constraint-Based Evaluation of Causal Reasoning in Vision-Language Modelscited by 0
  13. 2026Anytime-Valid LLM Leaderboards via Benchmark-weighted and Block-Factorized e-Processescited by 0
  14. 2026TraceDance: An Automated System for Building Agent Behavior Benchmarks from Real-World Agent Deployment Tracescited by 0
  15. 2026LiveOption: Evaluating LLM Agents in Structured Option Trading with Nonlinear Payoffscited by 0
  16. 2026OSWorld-Science: A Benchmark of Computer Use Agents for Learning and Using Scientific Softwarecited by 0
  17. 2026EviRover: Reinforcing Agentic Perception Beyond a Glancecited by 0
  18. 2026MOIRA: Mass-Oriented Indexing with Ragged Attention for Long-Context Decodingcited by 0
  19. 2026SPIN: Shadow Predictive Indexer for Sparse Attentioncited by 0
  20. 2026Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Positioncited by 0
  21. 2026What to Admit and How to Present: Governing Persistent Memory in LLM Agentscited by 0
  22. 2026PageWeaver: KV-Guided Query Unions for Sparse Attentioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokens
  2. 2026AB-Sparse: Sparse Attention with Adaptive Block Size for Accurate and Efficient Long-Context Inference
  3. 2026Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving
  4. 2025Compressed Convolutional Attention: Efficient Attention in a Compressed Latent Space
  5. 2025Hardware-Efficient Attention for Fast Decoding
  6. 2026SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference
  7. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs
  8. 2026CompactAttention: Accelerating Chunked Prefill with Block-Union KV Selection