mlsanityData to 14 Oct 2026

MoBA: Mixture of Block Attention for Long-Context LLMs

arXiv 2502.13189 ↗First posted 18 Feb 2025cs.LG cs.AI cs.CLcited by 198

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Scaling the effective context length is essential for advancing large language models (LLMs) toward artificial general intelligence (AGI). However, the quadratic increase in computational complexity inherent in traditional attention mechanisms presents a prohibitive overhead. Existing approaches either impose strongly biased structures, such as sink or window attention which are task-specific, or radically modify the attention mechanism into linear approximations, whose performance in complex reasoning tasks remains inadequately explored. In this work, we propose a solution that adheres to the ``less structure'' principle, allowing the model to determine where to attend autonomously, rather than introducing predefined biases. We introduce Mixture of Block Attention (MoBA), an innovative approach that applies the principles of Mixture of Experts (MoE) to the attention mechanism. This novel architecture demonstrates superior performance on long-context tasks while offering a key advantage: the ability to seamlessly transition between full and sparse attention, enhancing efficiency without the risk of compromising performance. MoBA has already been deployed to support Kimi's long-context requests and demonstrates significant advancements in efficient attention computation for LLMs. Our code is available at https://github.com/MoonshotAI/MoBA.

Key facts. Cited by 198 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 58 of its 68 references that are in the map. First posted on arXiv 18 Feb 2025.

Builds on

58 of 68 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025MiniMax-01: Scaling Foundation Models with Lightning Attention14 Jan
  2. 2024Qwen2.5 Technical Report19 Dec
  3. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs17 Oct
  4. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  5. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2020Longformer: The Long-Document Transformercited by 3,413
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  11. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  12. 2020Reformer: The Efficient Transformercited by 1,668
38 more
  1. 2020Rethinking Attention with Performerscited by 1,601
  2. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  3. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  4. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  5. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  6. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  7. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  8. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  9. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 652
  10. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  11. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  12. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  13. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 471
  14. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  15. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  16. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  17. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
  18. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  19. 2019Axial Attention in Multidimensional Transformerscited by 358
  20. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 308
  21. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  22. 2020Sparse Sinkhorn Attentioncited by 251
  23. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  24. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  25. 2018Online normalizer calculation for softmaxcited by 162
  26. 2019Blockwise Self-Attention for Long Document Understandingcited by 159
  27. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 159
  28. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 128
  29. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 122
  30. 2024Transformers are Multi-State RNNscited by 120
  31. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
  32. 2019Star-Transformercited by 94
  33. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  34. 2024Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Modelscited by 67
  35. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 63
  36. 2024Linearizing Large Language Modelscited by 51
  37. 2020GMAT: Global Memory Augmentation for Transformerscited by 36
  38. 2024LongHeads: Multi-Head Attention is Secretly a Long Context Processorcited by 16

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 83; 2026: 115.

198 papers.

Sort:

By area:

  1. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
  2. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 246
  3. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
  4. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 132
  5. 2025VSA: Faster Video Diffusion with Trainable Sparse Attentioncited by 104
  6. 2025Efficient Reasoning Models: A Surveycited by 95
  7. 2025Mixture of Contexts for Long Video Generationcited by 95
  8. 2025LongCat-Video Technical Reportcited by 85
  9. 2025Training-free and Adaptive Sparse Attention for Efficient Long Video Generationcited by 69
  10. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generationcited by 69
  11. 2025Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Modelcited by 55
  12. 2025MiniCPM4: Ultra-Efficient LLMs on End Devicescited by 52
38 more
  1. 2025End-to-End Training for Autoregressive Video Diffusion via Self-Resamplingcited by 52
  2. 2026MiniMax Sparse Attentioncited by 44
  3. 2025Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsificationcited by 37
  4. 2025VMoBA: Mixture-of-Block Attention for Video Diffusion Modelscited by 37
  5. 2025Training-Free Efficient Video Generation via Dynamic Token Carvingcited by 34
  6. 2025Understanding Transformer from the Perspective of Associative Memorycited by 34
  7. 2026LongCat-Flash-Thinking-2601 Technical Reportcited by 34
  8. 2025InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptationcited by 32
  9. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 31
  10. 2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 30
  11. 2025PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptationcited by 29
  12. 2025R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routingcited by 29
  13. 2025MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attentioncited by 26
  14. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 25
  15. 2025Block-Sparse Global Attention for Efficient Multi-View Geometry Transformerscited by 23
  16. 2025Efficient Attention Mechanisms for Large Language Models: A Surveycited by 22
  17. 2026SLA2: Sparse-Linear Attention with Learnable Routing and QATcited by 22
  18. 2025A Survey of Slow Thinking-based Reasoning LLMs using Reinforced Learning and Inference-time Scaling Lawcited by 19
  19. 2025Bidirectional Sparse Attention for Faster Video Diffusion Trainingcited by 19
  20. 2025MoGA: Mixture-of-Groups Attention for End-to-End Long Video Generationcited by 18
  21. 2026SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuningcited by 18
  22. 2026PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformerscited by 17
  23. 2026Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Evictioncited by 16
  24. 2025Comba: Improving Bilinear RNNs with Closed-loop Controlcited by 15
  25. 2026Memory Caching: RNNs with Growing Memorycited by 15
  26. 2025Hybrid Architectures for Language Models: Systematic Analysis and Design Insightscited by 14
  27. 2025FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Predictioncited by 13
  28. 2026HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attentioncited by 12
  29. 2026Hierarchical Sparse Attention Done Right: Toward Infinite Context Modelingcited by 12
  30. 2025MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systemscited by 11
  31. 2025SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimizationcited by 10
  32. 2025Rectified Sparse Attentioncited by 10
  33. 2025Which Heads Matter for Reasoning? RL-Guided KV Cache Compressioncited by 10
  34. 2026HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharingcited by 10
  35. 2026Speed3R: Sparse Feed-forward 3D Reconstruction Modelscited by 10
  36. 2026CausalCine: Real-Time Autoregressive Generation for Multi-Shot Video Narrativescited by 10
  37. 2025Hardware-aligned Hierarchical Sparse Attention for Efficient Long-term Memory Accesscited by 9
  38. 2025ACE-RL: Adaptive Constraint-Enhanced Reward for Long-form Generation Reinforcement Learningcited by 9

50 most cited of 198 All 198 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Attention-MoA: Enhancing Mixture-of-Agents via Inter-Agent Semantic Attention and Deep Residual Synthesis
  2. 2025Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing
  3. 2024Attention Heads of Large Language Models: A Survey
  4. 2026SPLA: Block Sparse Plus Linear Attention for Long Context Modeling
  5. 2022Mixture of Attention Heads: Selecting Attention Heads Per Token
  6. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving
  7. 2026Sliding-window beats linear attention
  8. 2026MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokens