mlsanityData to 13 Oct 2026

Falcon Mamba: The First Competitive Attention-free 7B Language Model

arXiv 2410.05355 ↗First posted 7 Oct 2024cs.CL cs.AIcited by 45

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

In this technical report, we present Falcon Mamba 7B, a new base large language model based on the novel Mamba architecture. Falcon Mamba 7B is trained on 5.8 trillion tokens with carefully selected data mixtures. As a pure Mamba-based model, Falcon Mamba 7B surpasses leading open-weight models based on Transformers, such as Mistral 7B, Llama3.1 8B, and Falcon2 11B. It is on par with Gemma 7B and outperforms models with different architecture designs, such as RecurrentGemma 9B and RWKV-v6 Finch 7B/14B. Currently, Falcon Mamba 7B is the best-performing Mamba model in the literature at this scale, surpassing both existing Mamba and hybrid Mamba-Transformer models, according to the Open LLM Leaderboard. Due to its architecture, Falcon Mamba 7B is significantly faster at inference and requires substantially less memory for long sequence generation. Despite recent studies suggesting that hybrid Mamba-Transformer models outperform pure architecture designs, we demonstrate that even the pure Mamba design can achieve similar, or even superior results compared to the Transformer and hybrid designs. We make the weights of our implementation of Falcon Mamba 7B publicly available on https://huggingface.co/tiiuae/falcon-mamba-7b, under a permissive license.

Key facts. Cited by 45 papers in the map (data through 13 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 40 of its 43 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

40 of 43 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Just read twice: closing the recall gap for recurrent language models7 Jul
  2. 2024An Empirical Study of Mamba-based Language Models12 Jun
  3. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling11 Jun
  4. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
  5. 2024Jamba: A Hybrid Transformer-Mamba Language Model28 Mar
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec

Other references · 34

  1. 2017Attention Is All You Needcited by 60,106
  2. 2024The Llama 3 Herd of Modelscited by 14,486
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,916
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,270
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,798
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,637
  8. 2020Longformer: The Long-Document Transformercited by 3,393
  9. 2023Mistral 7Bcited by 3,123
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,804
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,570
  12. 2024Qwen2 Technical Reportcited by 2,102
22 more
  1. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,799
  2. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,785
  3. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,191
  4. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,126
  5. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 908
  6. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 820
  7. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 760
  8. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 708
  9. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 693
  10. 2023The Falcon Series of Open Language Modelscited by 533
  11. 2023Llemma: An Open Language Model For Mathematicscited by 387
  12. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  13. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 233
  14. 2023MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoningcited by 200
  15. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  16. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 156
  17. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 119
  18. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 101
  19. 2024RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrievalcited by 62
  20. 2024RecurrentGemma: Moving Past Transformers for Efficient Open Language Modelscited by 61
  21. 2024Linearizing Large Language Modelscited by 51
  22. 2024Falcon2-11B Technical Reportcited by 12

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 4; 2025: 27; 2026: 14.

45 papers.

Sort:

By area:

  1. 2025STORM: Token-Efficient Long Video Understanding for Multimodal LLMscited by 48
  2. 2025LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargementcited by 39
  3. 2025Llamba: Scaling Distilled Recurrent Models for Efficient Language Processingcited by 35
  4. 2025Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformerscited by 35
  5. 2025Efficient Attention Mechanisms for Large Language Models: A Surveycited by 22
  6. 2024The Zamba2 Suite: Technical Reportcited by 20
  7. 2024Rethinking Uncertainty Estimation in LLMs: A Principled Single-Sequence Measurecited by 18
  8. 2025xLSTM 7B: A Recurrent LLM for Fast and Efficient Inferencecited by 17
  9. 2025LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routingcited by 17
  10. 2025Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanismcited by 15
  11. 2025ParaRNN: Unlocking Parallel Training of Nonlinear RNNs for Large Language Modelscited by 13
  12. 2025LoLA: Low-Rank Linear Attention With Sparse Cachingcited by 11
33 more
  1. 2024Truth or Mirage? Towards End-to-End Factuality Evaluation with LLM-Oasiscited by 8
  2. 2025Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Modelscited by 7
  3. 2025Addressing Pitfalls in the Evaluation of Uncertainty Estimation Methods for Natural Language Generationcited by 7
  4. 2025TransMamba: Fast Universal Architecture Adaption from Transformers to Mambacited by 6
  5. 2025Overflow Prevention Enhances Long-Context Recurrent LLMscited by 6
  6. 2025SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shotcited by 5
  7. 2025SpikingBrain: Spiking Brain-inspired Large Modelscited by 5
  8. 2025Understanding and Enhancing Mamba-Transformer Hybrids for Memory Recall and Language Modelingcited by 5
  9. 2026Convergent Evolution: How Different Language Models Learn Similar Number Representationscited by 5
  10. 2025Multilingual State Space Models for Structured Question Answering in Indic Languagescited by 3
  11. 2025On the Expressivity of Selective State-Space Layers: A Multivariate Polynomial Approachcited by 3
  12. 2025xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexitycited by 3
  13. 2025Recall with Reasoning: Chain-of-Thought Distillation for Mamba's Long-Context Memory and Extrapolationcited by 2
  14. 2025Emergence of Primacy and Recency Effect in Mamba: A Mechanistic Point of Viewcited by 2
  15. 2025Differential Mambacited by 2
  16. 2025Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMscited by 2
  17. 2025Predicate Renaming via Large Language Modelscited by 2
  18. 2025StreetMath: Study of LLMs' Approximation Behaviorscited by 2
  19. 2024PoM: Efficient Image and Video Generation with the Polynomial Mixercited by 1
  20. 2026Towards Large Model Feature Codingcited by 1
  21. 2025Beyond Semantics: How Temporal Biases Shape Retrieval in Transformer and State-Space Modelscited by 0
  22. 2026Scaling State-Space Models on Multiple GPUs with Tensor Parallelismcited by 0
  23. 2026PoM: A Linear-Time Replacement for Attention with the Polynomial Mixercited by 0
  24. 2026A Circuit, Not The Circuit: Non-Unique Causal Localisation of the Mamba-2 State Sinkcited by 0
  25. 2026Token-Operations-Oriented Inference Optimization Techniques for Large Modelscited by 0
  26. 2026Timesteps of Mamba Align with Human Reading Timescited by 0
  27. 2026GeoSelect: Spatial-Program Execution for Training-Free Referring Remote Sensing Image Segmentationcited by 0
  28. 2026An Exact Instrument for State Usage in Selective State-Space Models, and the Input-Driven Migration It Revealscited by 0
  29. 2026Locating and Steering Refusal Beyond Attentioncited by 0
  30. 2026Global Divergence, Local Convergence: Representation Geometry in SSMs and Transformerscited by 0
  31. 2026What Does Layer-Importance Reveal About Transformers and State-Space Models?cited by 0
  32. 2026LiAuto-MindViT: A Hybrid Vision Backbone with Adaptive Bidirectional Mambacited by 0
  33. 2026LogSig-SSM: Time-Series Modelling with Multi-Scale Log-Signature Compression for State-Space Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey of Mamba
  2. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Models
  3. 2025Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance
  4. 2025TransMamba: A Sequence-Level Hybrid Transformer-Mamba Language Model
  5. 2024Bi-Mamba: Towards Accurate 1-Bit State Space Models
  6. 2024ML-Mamba: Efficient Multi-Modal Large Language Model Utilizing Mamba-2
  7. 2025Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection
  8. 2024PackMamba: Efficient Processing of Variable-Length Sequences in Mamba training