mlsanityData to 7 Oct 2026

MiniMax-01: Scaling Foundation Models with Lightning Attention

arXiv 2501.08313 ↗First posted 14 Jan 2025cs.CL cs.CVcited by 157

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

We introduce MiniMax-01 series, including MiniMax-Text-01 and MiniMax-VL-01, which are comparable to top-tier models while offering superior capabilities in processing longer contexts. The core lies in lightning attention and its efficient scaling. To maximize computational capacity, we integrate it with Mixture of Experts (MoE), creating a model with 32 experts and 456 billion total parameters, of which 45.9 billion are activated for each token. We develop an optimized parallel strategy and highly efficient computation-communication overlap techniques for MoE and lightning attention. This approach enables us to conduct efficient training and inference on models with hundreds of billions of parameters across contexts spanning millions of tokens. The context window of MiniMax-Text-01 can reach up to 1 million tokens during training and extrapolate to 4 million tokens during inference at an affordable cost. Our vision-language model, MiniMax-VL-01 is built through continued training with 512 billion vision-language tokens. Experiments on both standard and in-house benchmarks show that our models match the performance of state-of-the-art models like GPT-4o and Claude-3.5-Sonnet while offering 20-32 times longer context window. We publicly release MiniMax-01 at https://github.com/MiniMax-AI.

Key facts. Cited by 157 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. First posted on arXiv 14 Jan 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 96; 2026: 61.

157 papers.

Sort:

By area:

  1. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  2. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  3. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  4. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  5. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  6. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  7. 2026MiMo-V2-Flash Technical Reportcited by 163
  8. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  9. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  10. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 91
  11. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoningcited by 75
  12. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
38 more
  1. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 69
  2. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  3. 2025It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimizationcited by 60
  4. 2025Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Modelscited by 53
  5. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
  6. 2025MoM: Linear Sequence Modeling with Mixture-of-Memoriescited by 44
  7. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 39
  8. 2025A Systematic Analysis of Hybrid Linear Attentioncited by 38
  9. 2025SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognitioncited by 35
  10. 2025Delta Decompression for MoE-based LLMs Compressioncited by 32
  11. 2025Liger: Linearizing Large Language Models to Gated Recurrent Structurescited by 32
  12. 2025Understanding Transformer from the Perspective of Associative Memorycited by 31
  13. 2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 28
  14. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 27
  15. 2025EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modescited by 26
  16. 2025LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contextscited by 25
  17. 2026ClawGuard: A Runtime Security Framework for Tool-Augmented LLM Agents Against Indirect Prompt Injectioncited by 25
  18. 2025Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidancecited by 23
  19. 2025Parallel Loop Transformer for Efficient Test-Time Computation Scalingcited by 23
  20. 2025Efficient Attention Mechanisms for Large Language Models: A Surveycited by 21
  21. 2025Rethinking Thinking Tokens: LLMs as Improvement Operatorscited by 20
  22. 2025Towards AI Search Paradigmcited by 19
  23. 2025Mid-Training of Large Language Models: A Surveycited by 18
  24. 2025A Survey on LLM Mid-Trainingcited by 18
  25. 2025GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?cited by 16
  26. 2026Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contextscited by 16
  27. 2025Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on Elementary School-Level Reasoning Problems?cited by 15
  28. 2025Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimizationcited by 14
  29. 2025iFlame: Interleaving Full and Linear Attention for Efficient Mesh Generationcited by 13
  30. 2025Hybrid Architectures for Language Models: Systematic Analysis and Design Insightscited by 13
  31. 2025Mixing Importance with Diversity: Joint Optimization for KV Cache Compression in Large Vision-Language Modelscited by 13
  32. 2025Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernelscited by 12
  33. 2025Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUscited by 12
  34. 2025Comba: Improving Bilinear RNNs with Closed-loop Controlcited by 12
  35. 2025Scaling Linear Attention with Sparse State Expansioncited by 12
  36. 2026Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scalingcited by 12
  37. 2026Memory Caching: RNNs with Growing Memorycited by 12
  38. 2026Text2CAD-Bench: A Benchmark for LLM-based Text-to-Parametric CAD Generationcited by 11

50 most cited of 157 All 157 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  2. 2026Measuring Maximum Activations in Open Large Language Models
  3. 2026The Model Knows Which Tokens Matter: Automatic Token Selection via Noise Gating
  4. 2026On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
  5. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  6. 2025MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism
  7. 2026CAPA: Contribution-Aware Pruning and FFN Approximation for Efficient Large Vision-Language Models
  8. 2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs