mlsanityData to 7 Oct 2026

Is Mamba Capable of In-Context Learning?

arXiv 2402.03170 ↗First posted 5 Feb 2024cs.LGcited by 45

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

State of the art foundation models such as GPT-4 perform surprisingly well at in-context learning (ICL), a variant of meta-learning concerning the learned ability to solve tasks during a neural network forward pass, exploiting contextual information provided as input to the model. This useful ability emerges as a side product of the foundation model's massive pretraining. While transformer models are currently the state of the art in ICL, this work provides empirical evidence that Mamba, a newly proposed state space model which scales better than transformers w.r.t. the input sequence length, has similar ICL capabilities. We evaluated Mamba on tasks involving simple function approximation as well as more complex natural language processing problems. Our results demonstrate that, across both categories of tasks, Mamba closely matches the performance of transformer models for ICL. Further analysis reveals that, like transformers, Mamba appears to solve ICL problems by incrementally optimizing its internal representations. Overall, our work suggests that Mamba can be an efficient alternative to transformers for ICL tasks involving long input sequences. This is an exciting finding in meta-learning and may enable generalizations of in-context learned AutoML algorithms (like TabPFN or Optformer) to long input sequences.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 49 of its 72 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

49 of 72 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Gated Linear Attention Transformers with Hardware-Efficient Training11 Dec
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  3. 2022Simplified State Space Layers for Sequence Modeling9 Aug
  4. 2022Diagonal State Spaces are as Effective as Structured State Spaces27 Mar
  5. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  8. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  9. 2020Rethinking Attention with Performerscited by 1,584
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  12. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
32 more
  1. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  2. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  3. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  4. 2023Large Language Models as Optimizerscited by 830
  5. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  6. 2022Transformers learn in-context by gradient descentcited by 714
  7. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  8. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  9. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  10. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
  11. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  12. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  13. 2024U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentationcited by 412
  14. 2021Random Feature Attentioncited by 350
  15. 2021Transformers Can Do Bayesian Inferencecited by 338
  16. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  17. 2023In-Context Learning Creates Task Vectorscited by 302
  18. 2023Function Vectors in Large Language Modelscited by 292
  19. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  20. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  21. 2022Long Range Language Modeling via Gated State Spacescited by 265
  22. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  23. 2019Blockwise Self-Attention for Long Document Understandingcited by 158
  24. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 115
  25. 2023Selective Structured State-Spaces for Long-Form Video Understandingcited by 105
  26. 2023ForecastPFN: Synthetically-Trained Zero-Shot Forecastingcited by 102
  27. 2021Finetuning Pretrained Transformers into RNNscited by 94
  28. 2024In-Context Language Learning: Architectures and Algorithmscited by 86
  29. 2022Towards Learning Universal Hyperparameter Optimizers with Transformerscited by 81
  30. 2023PFNs4BO: In-Context Learning for Bayesian Optimizationcited by 70
  31. 2023Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networkscited by 37
  32. 2023Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Abilitycited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 13; 2026: 5.

45 papers.

Sort:

By area:

  1. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  2. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
  3. 2024Is Mamba Effective for Time Series Forecasting?cited by 110
  4. 2024The Hidden Attention of Mamba Modelscited by 88
  5. 2026Mamba-3: Improved Sequence Modeling using State Space Principlescited by 76
  6. 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 73
  7. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  8. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  9. 2024Visual Mamba: A Survey and New Outlookscited by 41
  10. 2024Locating and Editing Factual Associations in Mambacited by 35
  11. 2024MTMamba: Enhancing Multi-Task Dense Scene Understanding by Mamba-Based Decoderscited by 22
  12. 2024Fine-grained Analysis of In-context Linear Estimation: Data, Architecture, and Beyondcited by 22
33 more
  1. 2024Mamba4Cast: Efficient Zero-Shot Time Series Forecasting with State Space Modelscited by 17
  2. 2024XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learningcited by 13
  3. 2024Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superpositioncited by 13
  4. 2024Mitigating Copy Bias in In-Context Learning through Neuron Pruningcited by 11
  5. 2025Gating is Weighting: Understanding Gated Linear Attention through In-context Learningcited by 11
  6. 2024MTMamba++: Enhancing Multi-Task Dense Scene Understanding via Mamba-Based Decoderscited by 10
  7. 2024Rethinking Mamba in Speech Processing by Self-Supervised Modelscited by 10
  8. 2025From Markov to Laplace: How Mamba In-Context Learns Markov Chainscited by 9
  9. 2025ms-Mamba: Multi-scale Mamba for Time-Series Forecastingcited by 7
  10. 2024Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modelingcited by 6
  11. 2024"Oh LLM, I'm Asking Thee, Please Give Me a Decision Tree": Zero-Shot Decision Tree Induction and Embedding with Large Language Modelscited by 6
  12. 2024Task Diversity Shortens the ICL Plateaucited by 5
  13. 2025Resona: Improving Context Copying in Linear Recurrence Models with Retrievalcited by 5
  14. 2024FACTS: A Factored State-Space Framework For World Modellingcited by 4
  15. 2025Mamba-Based Graph Convolutional Networks: Tackling Over-smoothing with Selective State Spacecited by 4
  16. 2025The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architecturescited by 3
  17. 2024Learning in the Recurrent State: Gradient Descent with Linear Recurrent Networkscited by 2
  18. 2025How Can Mamba Learn In Context with Outliers and Generalize Provably?cited by 2
  19. 2025Task Schema and Binding: A Double Dissociation Study of In-Context Learningcited by 2
  20. 2024EDCSSM: Edge Detection with Convolutional State Space Modelcited by 1
  21. 2024GAMformer: Bridging Tabular Foundation Models and Interpretable Machine Learningcited by 1
  22. 2024DeepOSets: Non-Autoregressive In-Context Learning with Permutation-Invariance Inductive Biascited by 1
  23. 2024Variable Aperture Bokeh Rendering via Customized Focal Plane Guidancecited by 1
  24. 2025Scaling Algorithm Distillation for Continuous Control with Mambacited by 1
  25. 2025Mamba Can Learn Low-Dimensional Targets In-Context via Test-Time Feature Learningcited by 1
  26. 2025Genomic Next-Token Predictors are In-Context Learnerscited by 1
  27. 2026Retrievit: In-context Retrieval Capabilities of Transformers, State Space Models, and Hybrid Architecturescited by 1
  28. 2026MamTra: A Hybrid Mamba-Transformer Backbone for Speech Synthesiscited by 1
  29. 2024Improving VTE Identification through Language Models from Radiology Reports: A Comparative Study of Mamba, Phi-3 Mini, and BERTcited by 0
  30. 2025What Neuroscience Can Teach AI About Learning in Continuously Changing Environmentscited by 0
  31. 2025From Text to Trajectories: GPT-2 as an ODE Solver via In-Contextcited by 0
  32. 2026Dendritic In-Context Learning in a Single-Layer Spiking Neural Networkcited by 0
  33. 2026On the Importance of Gating: Memorization vs. In-Context Learning in State Space Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks
  2. 2025Trained Mamba Emulates Online Gradient Descent in In-Context Linear Regression
  3. 2024A Survey of Mamba
  4. 2024An Empirical Study of Mamba-based Language Models
  5. 2024Can Custom Models Learn In-Context? An Exploration of Hybrid Architecture Performance on In-Context Learning Tasks
  6. 2025Probing In-Context Learning: Impact of Task Complexity and Model Architecture on Generalization and Efficiency
  7. 2025Mamba Modulation: On the Length Generalization of Mamba
  8. 2025Understanding In-Context Learning Beyond Transformers: An Investigation of State Space and Hybrid Architectures