mlsanityData to 7 Oct 2026

MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts

arXiv 2401.04081 ↗First posted 8 Jan 2024cs.LG cs.AI cs.CLcited by 60

On the map: Mixture-of-experts → MoE large language models

also in 1 more
Abstract (arXiv)

State Space Models (SSMs) have become serious contenders in the field of sequential modeling, challenging the dominance of Transformers. At the same time, Mixture of Experts (MoE) has significantly improved Transformer-based Large Language Models, including recent state-of-the-art open models. We propose that to unlock the potential of SSMs for scaling, they should be combined with MoE. We showcase this on Mamba, a recent SSM-based model that achieves remarkable performance. Our model, MoE-Mamba, outperforms both Mamba and baseline Transformer-MoE. In particular, MoE-Mamba reaches the same performance as Mamba in 2.35× fewer training steps while preserving the inference performance gains of Mamba against Transformer.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 7 of its 59 references that are in the map. First posted on arXiv 8 Jan 2024.

Builds on

7 of 59 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Efficient Large Scale Language Modeling with Mixtures of Experts20 Dec

Other references · 6

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  4. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  5. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  6. 2022What Makes Convolutional Models Great on Long Sequence Modeling?cited by 86

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 38; 2025: 15; 2026: 7.

60 papers.

Sort:

By area:

  1. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  2. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  3. 2024PointMamba: A Simple State Space Model for Point Cloud Analysiscited by 166
  4. 2024Is Mamba Effective for Time Series Forecasting?cited by 110
  5. 2024The Hidden Attention of Mamba Modelscited by 88
  6. 2024DRAMA: An Efficient End-to-end Motion Planner for Autonomous Driving with Mambacited by 87
  7. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  8. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  9. 2024A Survey on Visual Mambacited by 50
  10. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mambacited by 45
  11. 2024A Survey on Transformer Compressioncited by 44
  12. 2024SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domaincited by 39
38 more
  1. 2024BlackMamba: Mixture of Experts for State-Space Modelscited by 37
  2. 2024MambaLRP: Explaining Selective State Space Sequence Modelscited by 31
  3. 2024Learning Pattern-Specific Experts for Time Series Forecasting Under Patch-level Distribution Shiftcited by 23
  4. 2024Mamba Policy: Towards Efficient 3D Diffusion Policy with Hybrid Selective State Modelscited by 22
  5. 2024PointRWKV: Efficient RWKV-Like Model for Hierarchical Point Cloud Learningcited by 21
  6. 2025TREAD: Token Routing for Efficient Architecture-agnostic Diffusion Trainingcited by 21
  7. 2024DMM: Disparity-guided Multispectral Mamba for Oriented Object Detection in Remote Sensingcited by 19
  8. 2024Mambular: A Sequential Model for Tabular Deep Learningcited by 18
  9. 2024MambaTrack: A Simple Baseline for Multiple Object Tracking with State Space Modelcited by 18
  10. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  11. 2024CMViM: Contrastive Masked Vim Autoencoder for 3D Multi-modal Representation Learning for AD classificationcited by 15
  12. 2024Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RLcited by 15
  13. 2024SUM: Saliency Unification through Mamba for Visual Attention Modelingcited by 12
  14. 2025Spectral Informed Mamba for Robust Point Cloud Processingcited by 10
  15. 2024O-Mamba: O-shape State-Space Model for Underwater Image Enhancementcited by 9
  16. 20243DMambaComplete: Exploring Structured State Space Model for Point Cloud Completioncited by 8
  17. 2024Enhancing Online Continual Learning with Plug-and-Play State Space Model and Class-Conditional Mixture of Discretizationcited by 7
  18. 2024On the Efficiency of NLP-Inspired Methods for Tabular Deep Learningcited by 6
  19. 2024Efficient Fine-tuning of Audio Spectrogram Transformers via Soft Mixture of Adapterscited by 5
  20. 2024Sequential Order-Robust Mamba for Time Series Forecastingcited by 5
  21. 2025MoxE: Mixture of xLSTM Experts with Entropy-Aware Routing for Efficient Language Modelingcited by 4
  22. 2025Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projectioncited by 4
  23. 2024JambaTalk: Speech-Driven 3D Talking Head Generation Based on Hybrid Transformer-Mamba Modelcited by 3
  24. 2024The Implicit Bias of Structured State Space Models Can Be Poisoned With Clean Labelscited by 3
  25. 2025BioMamba: Leveraging Spectro-Temporal Embedding in Bidirectional Mamba for Enhanced Biosignal Classificationcited by 3
  26. 2025Technologies on Effectiveness and Efficiency: A Survey of State Spaces Modelscited by 2
  27. 2025Memba: Membrane-driven Parameter-Efficient Fine-Tuning for Mambacited by 2
  28. 2024SPMamba: State-space model is all you need in speech separationcited by 1
  29. 2024Mamba as Decision Maker: Exploring Multi-scale Sequence Modeling in Offline Reinforcement Learningcited by 1
  30. 2024Venturing into Uncharted Waters: The Navigation Compass from Transformer to Mambacited by 1
  31. 2024Demystifying the Token Dynamics of Deep Selective State Space Modelscited by 1
  32. 2024Exploring contextual modeling with linear complexity for point cloud segmentationcited by 1
  33. 2024Learning Mamba as a Continual Learner: Meta-Learning Selective State Space Models for Continual Learningcited by 1
  34. 2024MASV: Speaker Verification with Global and Local Context Mambacited by 1
  35. 2025Hybrid Tokenization Strategy for DNA Language Model using Byte Pair Encoding and K-MER Methodscited by 1
  36. 2025MixANT: Observation-dependent Memory Propagation for Stochastic Dense Action Anticipationcited by 1
  37. 2026MambaFormer: Token-Level Guided Routing Mixture-of-Experts for Accurate and Efficient Clinical Assistancecited by 1
  38. 2026Omni-directional attention mechanism based on Mamba for speech separationcited by 1

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024An Empirical Study of Mamba-based Language Models
  2. 2025Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Models
  3. 2024A Survey of Mamba
  4. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
  5. 2024Bi-Mamba: Towards Accurate 1-Bit State Space Models
  6. 2024Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models
  7. 2024MatMamba: A Matryoshka State Space Model
  8. 2024Sparse Mamba: Introducing Controllability, Observability, And Stability To Structural State Space Models