mlsanityData to 14 Oct 2026

Longhorn: State Space Models are Amortized Online Learners

arXiv 2407.14207 ↗First posted 19 Jul 2024cs.LGcited by 59

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

Modern large language models are built on sequence modeling via next-token prediction. While the Transformer remains the dominant architecture for sequence modeling, its quadratic decoding complexity in sequence length poses a major limitation. State-space models (SSMs) present a competitive alternative, offering linear decoding efficiency while maintaining parallelism during training. However, most existing SSMs rely on linear recurrence designs that appear somewhat ad hoc. In this work, we explore SSM design through the lens of online learning, conceptualizing SSMs as meta-modules for specific online learning problems. This approach links SSM design to formulating precise online learning objectives, with state transition rules derived from solving these objectives. Based on this insight, we introduce a novel deep SSM architecture, Longhorn, whose update resembles the closed-form solution for solving the online associative recall problem. Our experimental results show that Longhorn outperforms state-of-the-art SSMs, including the Mamba model, on standard sequence modeling benchmarks, language modeling, and vision tasks. Specifically, Longhorn achieves a 1.8x improvement in sample efficiency compared to Mamba, and can extrapolate over contexts that are up to 16x longer during inference.

Key facts. Cited by 59 papers in the map (data through 14 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 47 of its 65 references that are in the map. First posted on arXiv 19 Jul 2024.

Builds on

47 of 65 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Just read twice: closing the recall gap for recurrent language models7 Jul
  2. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden States5 Jul
  3. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling11 Jun
  4. 2024Simple linear attention language models balance the recall-throughput tradeoff28 Feb
  5. 2023Gated Linear Attention Transformers with Hardware-Efficient Training11 Dec
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  7. 2022Simplified State Space Layers for Sequence Modeling9 Aug
  8. 2022Diagonal State Spaces are as Effective as Structured State Spaces27 Mar
  9. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  11. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  12. 2020Rethinking Attention with Performerscited by 1,601
26 more
  1. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  2. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  3. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  4. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,056
  5. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  6. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  7. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  8. 2016Language Modeling with Gated Convolutional Networkscited by 702
  9. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  10. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  11. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 499
  12. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  13. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  14. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
  15. 2024xLSTM: Extended Long Short-Term Memorycited by 327
  16. 2022Long Range Language Modeling via Gated State Spacescited by 265
  17. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 242
  18. 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 204
  19. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 159
  20. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 127
  21. 2023Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakescited by 111
  22. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 102
  23. 2023Uncovering mesa-optimization algorithms in Transformerscited by 100
  24. 2022What Makes Convolutional Models Great on Long Sequence Modeling?cited by 87
  25. 2022Pretraining Without Attentioncited by 51
  26. 2022DOM-LM: Learning Generalizable Representations for HTML Documentscited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 21; 2026: 32.

59 papers.

Sort:

By area:

  1. 2024Titans: Learning to Memorize at Test Timecited by 328
  2. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
  3. 2025TTT3R: 3D Reconstruction as Test-Time Trainingcited by 129
  4. 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 76
  5. 2025DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Productscited by 75
  6. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 74
  7. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 72
  8. 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 66
  9. 2025It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimizationcited by 64
  10. 2025MesaNet: Sequence Modeling by Locally Optimal Test-Time Trainingcited by 47
  11. 2026Gated DeltaNet-2: Decoupling Erase and Write in Linear Attentioncited by 33
  12. 2025Understanding and Improving Length Generalization in Recurrent Modelscited by 23
38 more
  1. 2025Efficient Attention Mechanisms for Large Language Models: A Surveycited by 22
  2. 2025Lattice: Learning to Efficiently Compress the Memorycited by 19
  3. 2024Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothingcited by 18
  4. 2026Memory Caching: RNNs with Growing Memorycited by 15
  5. 2026Test-Time Training with KV Binding Is Secretly Linear Attentioncited by 13
  6. 2025Gated KalmaNet: A Fading Memory Layer Through Test-Time Ridge Regressioncited by 11
  7. 2025Mechanistic evaluation of Transformers and state space modelscited by 9
  8. 2025RAD: Redundancy-Aware Distillation for Hybrid Models via Self-Speculative Decodingcited by 8
  9. 2025TNT: Improving Chunkwise Training for Test-Time Memorizationcited by 8
  10. 2026Kalman Linear Attention: Parallel Bayesian Filtering For Efficient Language Modelling and State Trackingcited by 8
  11. 2026Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrencescited by 7
  12. 2024Learning Memory Mechanisms for Decision Making through Demonstrationscited by 6
  13. 2025RoboSSM: Scalable In-context Imitation Learning via State-Space Modelscited by 5
  14. 2025SAS: Simulated Attention Scorecited by 4
  15. 2025Local Linear Attention: An Optimal Interpolation of Linear and Softmax Attention For Test-Time Regressioncited by 4
  16. 2026MDN: Parallelizing Stepwise Momentum for Delta Linear Attentioncited by 4
  17. 2024Learning in the Recurrent State: Gradient Descent with Linear Recurrent Networkscited by 3
  18. 2026Fast Spatial Memory with Elastic Test-Time Trainingcited by 3
  19. 2026Continuous Latent Contexts Enable Efficient Online Learning in Transformerscited by 3
  20. 2026MuonSSM: Orthogonalizing State Space Models for Sequence Modelingcited by 3
  21. 2026Maglev: Sliding Recurrent Memorycited by 3
  22. 2025Untangling Component Imbalance in Hybrid Linear Attention Conversion Methodscited by 2
  23. 2026Learning to Remember, Learn, and Forget in Attention-Based Modelscited by 2
  24. 2026Tapered Language Modelscited by 2
  25. 2026MARCH: Scaling Recurrent Memory with Content-Routed State Anchorscited by 2
  26. 2024Learning Mamba as a Continual Learner: Meta-Learning Selective State Space Models for Continual Learningcited by 1
  27. 2025Hydra: Dual Exponentiated Memory for Multivariate Time Series Analysiscited by 1
  28. 2026A State-Transition Framework for Efficient LLM Reasoningcited by 1
  29. 2026Beyond Test-Time Memory: State-Space Optimal Control for LLM Reasoningcited by 1
  30. 2026FG2-GDN: Enhancing Long-Context Gated Delta Networks with Doubly Fine-Grained Controlcited by 1
  31. 2026Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillationcited by 1
  32. 2026SHiPPO: Recurrent Memory with Transported Polynomial Projectionscited by 1
  33. 2026The Key to Going Linear: Analysis-Driven Transformer Linearizationcited by 1
  34. 2026Proteus: Incremental Memory Activation for Long-Context Sequence Modelingcited by 1
  35. 2026Safin-1: Safety from Within through Memory-Native State Evolutioncited by 1
  36. 2026RunningTensor: Generalizing Linear Attention to Higher-Order Recurrent Statescited by 1
  37. 2025Enhancing Linear Attention with Residual Learningcited by 0
  38. 2026Memory by Design: Bayesian Layers for Sequence Modelingcited by 0

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mamba
  2. 2025LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargement
  3. 2023State space models can express n-gram languages
  4. 2025Characterizing Mamba's Selective Memory using Auto-Encoders
  5. 2025MS-SSM: A Multi-Scale State Space Model for Efficient Sequence Modeling
  6. 2024BlackMamba: Mixture of Experts for State-Space Models
  7. 2026Mamba-3: Improved Sequence Modeling using State Space Principles
  8. 2026Harmonic: Hierarchical State Space Models for Efficient Long-Context Language Modeling