mlsanityData to 7 Oct 2026

BlackMamba: Mixture of Experts for State-Space Models

arXiv 2402.01771 ↗First posted 1 Feb 2024cs.CL cs.AI cs.DCcited by 37

On the map: Mixture-of-experts → MoE large language models

also in 1 more
Abstract (arXiv)

State-space models (SSMs) have recently demonstrated competitive performance to transformers at large-scale language modeling benchmarks while achieving linear time and memory complexity as a function of sequence length. Mamba, a recently released SSM model, shows impressive performance in both language modeling and long sequence processing tasks. Simultaneously, mixture-of-expert (MoE) models have shown remarkable performance while significantly reducing the compute and latency costs of inference at the expense of a larger memory footprint. In this paper, we present BlackMamba, a novel architecture that combines the Mamba SSM with MoE to obtain the benefits of both. We demonstrate that BlackMamba performs competitively against both Mamba and transformer baselines, and outperforms in inference and training FLOPs. We fully train and open-source 340M/1.5B and 630M/2.8B BlackMamba models on 300B tokens of a custom dataset. We show that BlackMamba inherits and combines both of the benefits of SSM and MoE architectures, combining linear-complexity generation from SSM with cheap and fast inference from MoE. We release all weights, checkpoints, and inference code open-source. Inference code at: https://github.com/Zyphra/BlackMamba

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 36 of its 44 references that are in the map. First posted on arXiv 1 Feb 2024.

Builds on

36 of 44 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Mixtral of Experts8 Jan
  2. 2024MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts8 Jan
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun

Other references · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2023Mistral 7Bcited by 3,065
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
21 more
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  2. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  5. 2020GLU Variants Improve Transformercited by 1,724
  6. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  7. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  8. 2022A Generalist Agentcited by 802
  9. 2023StarCoder: may the source be with you!cited by 790
  10. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  11. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  12. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  13. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  14. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  15. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  16. 2023Llemma: An Open Language Model For Mathematicscited by 385
  17. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  18. 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 188
  19. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  20. 2023What's In My Big Data?cited by 129
  21. 2023Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecastingcited by 102

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 20; 2025: 10; 2026: 7.

37 papers.

Sort:

By area:

  1. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
  2. 2024The Hidden Attention of Mamba Modelscited by 88
  3. 2024ZigMa: A DiT-style Zigzag Mamba Diffusion Modelcited by 84
  4. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  5. 2024A Survey on Visual Mambacited by 50
  6. 2024Understanding the Performance and Estimating the Cost of LLM Fine-Tuningcited by 48
  7. 2024A Survey of Mambacited by 39
  8. 2024MambaLRP: Explaining Selective State Space Sequence Modelscited by 31
  9. 2024Learning Pattern-Specific Experts for Time Series Forecasting Under Patch-level Distribution Shiftcited by 23
  10. 2024PointRWKV: Efficient RWKV-Like Model for Hierarchical Point Cloud Learningcited by 21
  11. 2024The Zamba2 Suite: Technical Reportcited by 20
  12. 2024CMViM: Contrastive Masked Vim Autoencoder for 3D Multi-modal Representation Learning for AD classificationcited by 15
25 more
  1. 2025FlashMoE: Fast Distributed MoE in a Single Kernelcited by 13
  2. 2024Zyda-2: a 5 Trillion Token High-Quality Datasetcited by 12
  3. 2024Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulationcited by 11
  4. 2024Provable Benefits of Complex Parameterizations for Structured State Space Modelscited by 9
  5. 2024Enhancing Online Continual Learning with Plug-and-Play State Space Model and Class-Conditional Mixture of Discretizationcited by 7
  6. 2025Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsitycited by 7
  7. 2024Sequential Order-Robust Mamba for Time Series Forecastingcited by 5
  8. 2025Continual Pre-training of MoEs: How robust is your router?cited by 5
  9. 2025Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projectioncited by 4
  10. 2024Mamba State-Space Models Are Lyapunov-Stable Learnerscited by 3
  11. 2025CymbaDiff: Structured Spatial Diffusion for Sketch-based 3D Semantic Urban Scene Generationcited by 3
  12. 2025Technologies on Effectiveness and Efficiency: A Survey of State Spaces Modelscited by 2
  13. 2025Online Mixture of Experts: No-Regret Learning for Optimal Collective Decision-Makingcited by 2
  14. 2026Hidden State Poisoning Attacks against Mamba-based Language Modelscited by 2
  15. 2026Efficient Continual Learning in Language Models via Thalamically Routed Cortical Columnscited by 2
  16. 2024Mamba as Decision Maker: Exploring Multi-scale Sequence Modeling in Offline Reinforcement Learningcited by 1
  17. 2024Venturing into Uncharted Waters: The Navigation Compass from Transformer to Mambacited by 1
  18. 2025MixANT: Observation-dependent Memory Propagation for Stochastic Dense Action Anticipationcited by 1
  19. 2026WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardwarecited by 1
  20. 2026On the Recall Scaling Laws in Mamba: A Theoretical and Mechanistic Study via Hashingcited by 1
  21. 2025Adjoint sharding for very long context training of state space modelscited by 0
  22. 2025Parameter Aware Mamba Model for Multi-task Dense Predictioncited by 0
  23. 2026Swimba: Switch Mamba Model Scales State Space Modelscited by 0
  24. 2026A Unified Mamba--MoE Surrogate for Closed-Loop Simulation and Measurement-Window Forecasting of Inverter Transientscited by 0
  25. 2026More Experts, Worse Dynamics: Inverse Scaling and Spectral Bias in Mixture-of-Experts State-Space Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024An Empirical Study of Mamba-based Language Models
  2. 2024Bi-Mamba: Towards Accurate 1-Bit State Space Models
  3. 2026Mamba-3: Improved Sequence Modeling using State Space Principles
  4. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  5. 2024A Survey on Mixture of Experts in Large Language Models
  6. 2025Mixture of Experts in Large Language Models
  7. 2024ReMamba: Equip Mamba with Effective Long-Sequence Modeling
  8. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline