mlsanityData to 7 Oct 2026

How Can Mamba Learn In Context with Outliers and Generalize Provably?

arXiv 2510.00399 ↗First posted 1 Oct 2025cs.LGcited by 2

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

The Mamba model has gained significant attention for its computational advantages over Transformer-based models, while achieving comparable performance across a wide range of language tasks. Like Transformers, Mamba exhibits in-context learning (ICL) capabilities, i.e., making predictions for new tasks based on a prompt containing input-label pairs and a query, without requiring fine-tuning. Despite its empirical success, the theoretical understanding of Mamba remains limited, largely due to the nonlinearity introduced by its gating mechanism. To the best of our knowledge, this paper presents the first theoretical analysis of the training dynamics of a one-layer Mamba model, which consists of a linear attention component followed by a nonlinear gating layer, and its ICL generalization on unseen binary classification tasks, even when the prompt includes additive outliers. Our analysis shows that Mamba leverages the linear attention layer to select informative context examples and uses the nonlinear gating layer to suppress the influence of outliers. By establishing and comparing to the analysis of linear Transformers under the same setting, we show that although Mamba may require more training iterations to converge, it maintains accurate predictions even when the proportion of outliers exceeds the threshold that a linear Transformer can tolerate. These theoretical findings are supported by empirical experiments.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 82 of its 97 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

82 of 97 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025From Markov to Laplace: How Mamba In-Context Learns Markov Chains14 Feb
  2. 2024Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing31 Dec
  3. 2024An Empirical Study of Mamba-based Language Models12 Jun
  4. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
  5. 2024The Hidden Attention of Mamba Models3 Mar
  6. 2024Simple linear attention language models balance the recall-throughput tradeoff28 Feb
  7. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks6 Feb
  8. 2024Is Mamba Capable of In-Context Learning?5 Feb
  9. 2023Gated Linear Attention Transformers with Hardware-Efficient Training11 Dec
  10. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  11. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 71

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  6. 2022In-context Learning and Induction Headscited by 930
  7. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  8. 2022Transformers learn in-context by gradient descentcited by 714
  9. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  10. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  11. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  12. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
38 more
  1. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  2. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  3. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  4. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  5. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  6. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  7. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  8. 2023Poisoning Language Models During Instruction Tuningcited by 254
  9. 2024MambaVision: A Hybrid Mamba-Transformer Vision Backbonecited by 222
  10. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
  11. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  12. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
  13. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  14. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 115
  15. 2023In-Context Convergence of Transformerscited by 112
  16. 2024Demystify Mamba in Vision: A Linear Attention Perspectivecited by 101
  17. 2017Recovery Guarantees for One-hidden-layer Neural Networkscited by 94
  18. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 93
  19. 2022Vision Transformers provably learn spatial structurecited by 92
  20. 2023Backdoor Attacks for In-Context Learning with Language Modelscited by 92
  21. 2020Learning Parities with Neural Networkscited by 77
  22. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  23. 2024Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimalitycited by 71
  24. 2022A Theoretical Analysis on Feature Learning in Neural Networks: Emergence from Inputs and Advantage over Fixed Featurescited by 49
  25. 2024Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learningcited by 47
  26. 2024Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformerscited by 46
  27. 2024Understanding Factual Recall in Transformers via Associative Memoriescited by 45
  28. 2024Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscapecited by 43
  29. 2023Hijacking Large Language Models via Adversarial In-Context Learningcited by 41
  30. 2023What can a Single Attention Layer Learn? A Study Through the Random Features Lenscited by 39
  31. 2025When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformerscited by 38
  32. 2024In-Context Learning with Representations: Contextual Generalization of Trained Transformerscited by 36
  33. 2023Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networkscited by 35
  34. 2024How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?cited by 35
  35. 2024Pretrained transformer efficiently learns low-dimensional target functions in-contextcited by 29
  36. 2024Parallel Structures in Pre-training Data Yield In-Context Learningcited by 26
  37. 2022Generalization Guarantee of Training Graph Convolutional Networks with Graph Topology Samplingcited by 23
  38. 2018Guaranteed Recovery of One-Hidden-Layer Neural Networks via Cross Entropycited by 22

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Mamba Can Learn Low-Dimensional Targets In-Context via Test-Time Feature Learningcited by 1
  2. 2026Transformers Learn the Optimal DDPM Denoiser for Multi-Token GMMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Mamba-PTQ: Outlier Channels in Recurrent Large Language Models
  2. 2025Trained Mamba Emulates Online Gradient Descent in In-Context Linear Regression
  3. 2024A Survey of Mamba
  4. 2026A Theoretical Analysis of Mamba's Training Dynamics: Filtering Relevant Features for Generalization in State Space Models
  5. 2025Probing In-Context Learning: Impact of Task Complexity and Model Architecture on Generalization and Efficiency
  6. 2025LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargement
  7. 2025Mamba Modulation: On the Length Generalization of Mamba
  8. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mamba