SHiPPO: Recurrent Memory with Transported Polynomial Projections
Not on the map. Why?
Abstract (arXiv)
HiPPO gives recurrent states memory semantics as coefficients of online polynomial projections, but in fixed channel coordinates. Modern selective SSMs, by contrast, rely on token-dependent control and channel interaction. We introduce SHiPPO (Sylvester HiPPO), a transported projection-memory prior that lifts HiPPO coefficient memories into a moving channel frame. For any fixed or realized right-transport path, SHiPPO transports the approximation family and channel metric together; conditional on that path, the state is ordinary HiPPO in a tied moving frame and follows Sylvester coefficient dynamics, preserving the left online-memory operator while adding right-action transport. For selective-SSM execution, we derive a restricted group-local realization with controller-compatible right actions, exponential-adjusted updates, exact block-affine scan, and recurrent decoding. We also give a simultaneous-reducibility criterion identifying when right transports collapse to static mixing plus independent scalar or blockwise banks. Controlled diagnostics show that larger current-token write rank improves ordinary prediction error but cannot recover order-sensitive changes to already-written memory; transported-memory variants recover this signal, which disappears when the transport pathway is removed. A finite-field associative-recall diagnostic with interleaved bindings, operations, and queries provides complementary autoregressive evidence while leaving the preferred right-action realization open. Taken together, these results support SHiPPO as a mechanistically grounded transported-memory prior, with evidence focused on memory mechanisms rather than broad sequence-modeling dominance.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 62 of its 76 references that are in the map. First posted on arXiv 3 Jul 2026.
Builds on
62 of 76 references.
References · 62
- 2017Attention Is All You Needcited by 59,932
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
- 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
- 2018Neural Ordinary Differential Equationscited by 3,733
- 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
- 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
- 2022Simplified State Space Layers for Sequence Modelingcited by 776
- 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
- 2020Neural Controlled Differential Equations for Irregular Time Seriescited by 501
- 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
- 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
- 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
38 more
- 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
- 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
- 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
- 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
- 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
- 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
- 2019Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systemscited by 203
- 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 188
- 2024The Illusion of State in State-Space Modelscited by 181
- 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
- 2020Liquid Time-constant Networkscited by 160
- 2020Model-Based Deep Learningcited by 136
- 2022Liquid Structural State-Space Modelscited by 134
- 2017Parallelizing Linear Recurrent Neural Nets Over Sequence Lengthcited by 126
- 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
- 2022How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projectionscited by 121
- 2019AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networkscited by 112
- 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
- 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvaluescited by 97
- 2020Lipschitz Recurrent Neural Networkscited by 88
- 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
- 2020Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependenciescited by 81
- 2026Mamba-3: Improved Sequence Modeling using State Space Principlescited by 76
- 2024Theoretical Foundations of Deep Selective State-Space Modelscited by 73
- 2024Longhorn: State Space Models are Amortized Online Learnerscited by 58
- 2024The Expressive Capacity of State Space Models: A Formal Language Perspectivecited by 45
- 2023State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memorycited by 38
- 2025MesaNet: Sequence Modeling by Locally Optimal Test-Time Trainingcited by 37
- 2024Oscillatory State-Space Modelscited by 35
- 2023StableSSM: Alleviating the Curse of Memory in State-space Models through Stable Reparameterizationcited by 29
- 2024Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networkscited by 26
- 2023Robustifying State-space Models for Long Sequences via Approximate Diagonalizationcited by 18
- 2024Tuning Frequency Bias of State Space Modelscited by 16
- 2025Structured Linear CDEs: Maximally Expressive and Parallel-in-Time Sequence Modelscited by 16
- 2024Mimetic Initialization Helps State Space Models Learn to Recallcited by 15
- 2024On the Expressiveness and Length Generalization of Selective State-Space Models on Regular Languagescited by 15
- 2025Structured Sparse Transition Matrices to Enable State Tracking in State-Space Modelscited by 15
- 2024HOPE for a Robust Parameterization of Long-memory State Space Modelscited by 11
50 most cited of 62 All 62 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Naju: A Native Discrete State-Space Model with Independent Retention and Writing for Long-Sequence Memory
- 2026HiPPO Zoo: Explicit Memory Mechanisms for Interpretable State Space Models
- 2024Mathematical Formalism for Memory Compression in Selective State Space Models
- 2026MaRK: Markov-adapted Recurrent Kernels for Dynamic Operator Conditioning in State Space Models
- 2026Fast Weight Attention for Continual Learning
- 2024HiPPO-Prophecy: State-Space Models can Provably Learn Dynamical Systems in Context
- 2026TCP-SSM: Efficient Vision State Space Models with Token-Conditioned Poles
- 2026Length-Independent State Tracking Under a Parallel Scan