mlsanityData to 14 Oct 2026

Understanding Mamba-3 Through Oscillatory Dynamics

arXiv 2610.13341 ↗First posted 9 Oct 2026cs.LGcited by 0

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

Mamba-3 introduces input-dependent rotations into recurrent state updates, but their contribution beyond state retention remains unclear. We interpret these rotations as underdamped second-order dynamics, providing a framework for investigating the role of oscillation in sequence modeling. Controlled comparisons show that underdamped dynamics outperform non-oscillatory damping regimes on state-tracking tasks, while language-model pretraining demonstrates benefits for long-context prediction and length generalization. Interventions in pretrained models further establish the importance of learned phase, and state analysis suggests that rotation regulates how retained information combines with incoming updates. Taken together, these findings identify underdamped dynamics as a useful mechanism for recurrent computation and suggest that oscillation supports long-context processing by regulating how information accumulates.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 40 of its 48 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

40 of 48 references.

Closest ancestors · 10

Same direction, newest first

  1. 2026Mamba-3: Improved Sequence Modeling using State Space Principles16 Mar
  2. 2025Mamba Modulation: On the Length Generalization of Mamba23 Sep
  3. 2025LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargement22 Apr
  4. 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues19 Nov
  5. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mamba20 Jun
  6. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
  7. 2024Locating and Editing Factual Associations in Mamba4 Apr
  8. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  9. 2022Simplified State Space Layers for Sequence Modeling9 Aug
  10. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  11. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  12. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
18 more
  1. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  2. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  3. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  4. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  5. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 424
  6. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  7. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  8. 2015Unitary Evolution Recurrent Neural Networkscited by 294
  9. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 200
  10. 2016Full-Capacity Unitary Recurrent Neural Networkscited by 94
  11. 2020Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependenciescited by 82
  12. 2016Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNscited by 65
  13. 2024Oscillatory State-Space Modelscited by 39
  14. 2023Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvaluescited by 36
  15. 2025Learning to Dissipate Energy in Oscillatory State-Space Modelscited by 9
  16. 2025Selective Rotary Position Embeddingcited by 9
  17. 2026MDN: Parallelizing Stepwise Momentum for Delta Linear Attentioncited by 4
  18. 2026Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attentioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Emergence of Primacy and Recency Effect in Mamba: A Mechanistic Point of View
  2. 2025MemMamba: Rethinking Memory Patterns in State Space Model
  3. 2026Motif-Mamba: network motif improved mamba for long-range sequence modeling
  4. 2026Timesteps of Mamba Align with Human Reading Times
  5. 2025Understanding Input Selectivity in Mamba: Impact on Approximation Power, Memorization, and Associative Recall Capacity
  6. 2026Temporal Context Reinstatement Drives Episodic-Like Order Memory in Long-Context Language Models
  7. 2026Rethinking State Tracking in Recurrent Models Through Error Control Dynamics
  8. 2026On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models