mlsanityData to 7 Oct 2026

Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks

arXiv 2402.04248 ↗First posted 6 Feb 2024cs.LGcited by 104

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

State-space models (SSMs), such as Mamba (Gu & Dao, 2023), have been proposed as alternatives to Transformer networks in language modeling, by incorporating gating, convolutions, and input-dependent token selection to mitigate the quadratic cost of multi-head attention. Although SSMs exhibit competitive performance, their in-context learning (ICL) capabilities, a remarkable emergent property of modern language models that enables task execution without parameter optimization, remain underexplored compared to Transformers. In this study, we evaluate the ICL performance of SSMs, focusing on Mamba, against Transformer models across various tasks. Our results show that SSMs perform comparably to Transformers in standard regression ICL tasks, while outperforming them in tasks like sparse parity learning. However, SSMs fall short in tasks involving non-standard retrieval functionality. To address these limitations, we introduce a hybrid model, MambaFormer, that combines Mamba with attention blocks, surpassing individual models in tasks where they struggle independently. Our findings suggest that hybrid architectures offer promising avenues for enhancing ICL in language models.

Key facts. Cited by 104 papers in the map (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 50 of its 60 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

50 of 60 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Gated Linear Attention Transformers with Hardware-Efficient Training11 Dec
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  3. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2022Emergent Abilities of Large Language Modelscited by 2,483
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  10. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  11. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  12. 2022In-context Learning and Induction Headscited by 930
35 more
  1. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  2. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  3. 2022Transformers learn in-context by gradient descentcited by 714
  4. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  5. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  6. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  7. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  8. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  9. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  10. 2021MetaICL: Learning to Learn In Contextcited by 450
  11. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  12. 2023Scaling Data-Constrained Language Modelscited by 391
  13. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  14. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  15. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  16. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
  17. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  18. 2022Long Range Language Modeling via Gated State Spacescited by 265
  19. 2022Transformers Learn Shortcuts to Automatacited by 260
  20. 2023Looped Transformers as Programmable Computerscited by 224
  21. 2022Neural Networks and the Chomsky Hierarchycited by 214
  22. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 198
  23. 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 188
  24. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
  25. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 168
  26. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 135
  27. 2021An Attention Free Transformercited by 111
  28. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  29. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  30. 2024In-Context Language Learning: Architectures and Algorithmscited by 86
  31. 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 74
  32. 2023Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functionscited by 62
  33. 2022Efficient Long Sequence Modeling via State Space Augmented Transformercited by 37
  34. 2016Fast Learning Requires Good Memory: A Time-Space Lower Bound for Parity Learningcited by 32
  35. 2023Block-State Transformerscited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 51; 2025: 35; 2026: 18.

104 papers.

Sort:

By area:

  1. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  2. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  3. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  4. 2024An Empirical Study of Mamba-based Language Modelscited by 177
  5. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  6. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
  7. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 100
  8. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
  9. 2024The Hidden Attention of Mamba Modelscited by 88
  10. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 69
  11. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  12. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
38 more
  1. 2024Visual Mamba: A Survey and New Outlookscited by 41
  2. 2024A Survey of Mambacited by 39
  3. 2025A Systematic Analysis of Hybrid Linear Attentioncited by 38
  4. 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 29
  5. 2025Next-Latent Prediction Transformers Learn Compact World Modelscited by 29
  6. 2024PoinTramba: A Hybrid Transformer-Mamba Framework for Point Cloud Analysiscited by 26
  7. 2024Dimba: Transformer-Mamba Diffusion Modelscited by 26
  8. 2024Fine-grained Analysis of In-context Linear Estimation: Data, Architecture, and Beyondcited by 22
  9. 2024PointRWKV: Efficient RWKV-Like Model for Hierarchical Point Cloud Learningcited by 21
  10. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 21
  11. 2024The Zamba2 Suite: Technical Reportcited by 20
  12. 2024Mamba4Cast: Efficient Zero-Shot Time Series Forecasting with State Space Modelscited by 17
  13. 2024Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothingcited by 17
  14. 2024An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encodingcited by 16
  15. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  16. 2024DyG-Mamba: Continuous State Space Modeling on Dynamic Graphscited by 15
  17. 2025Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanismcited by 15
  18. 2025Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Modelscited by 15
  19. 2024Training Nonlinear Transformers for Chain-of-Thought Inference: A Theoretical Generalization Analysiscited by 14
  20. 2024XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learningcited by 13
  21. 2024Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superpositioncited by 13
  22. 2025Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routingcited by 13
  23. 2025Mamba Modulation: On the Length Generalization of Mambacited by 13
  24. 2025Hybrid Architectures for Language Models: Systematic Analysis and Design Insightscited by 13
  25. 2024MLPs Learn In-Context on Regression and Classification Taskscited by 12
  26. 2024ReMamba: Equip Mamba with Effective Long-Sequence Modelingcited by 12
  27. 2024Mitigating Copy Bias in In-Context Learning through Neuron Pruningcited by 11
  28. 2024Parameter-Efficient Fine-Tuning of State Space Modelscited by 11
  29. 2024DiMSUM: Diffusion Mamba -- A Scalable and Unified Spatial-Frequency Method for Image Generationcited by 11
  30. 2025Gating is Weighting: Understanding Gated Linear Attention through In-context Learningcited by 11
  31. 2024How Well Can a Long Sequence Model Model Long Sequences? Comparing Architechtural Inductive Biases on Long-Context Abilitiescited by 9
  32. 2024A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contextscited by 9
  33. 2025From Markov to Laplace: How Mamba In-Context Learns Markov Chainscited by 9
  34. 2025Towards Understanding Visual Grounding in Visual Language Modelscited by 9
  35. 2024Integration of Mamba and Transformer -- MAT for Long-Short Range Time Series Forecasting with Application to Weather Dynamicscited by 7
  36. 2024Stuffed Mamba: Oversized States Lead to the Inability to Forgetcited by 7
  37. 2024Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modelingcited by 6
  38. 2024Heracles: A Hybrid SSM-Transformer Model for High-Resolution Image and Time-Series Analysiscited by 5

50 most cited of 104 All 104 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Is Mamba Capable of In-Context Learning?
  2. 2025LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargement
  3. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
  4. 2025TransMamba: A Sequence-Level Hybrid Transformer-Mamba Language Model
  5. 2025Trained Mamba Emulates Online Gradient Descent in In-Context Linear Regression
  6. 2024How Effective are State Space Models for Machine Translation?
  7. 2025MaTVLM: Hybrid Mamba-Transformer for Efficient Vision-Language Modeling
  8. 2024MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts