mlsanityData to 14 Oct 2026

Numerical Analysis of HiPPO-LegS ODE for Deep State Space Models

arXiv 2412.08595 ↗First posted 11 Dec 2024math.NA cs.LG cs.NAcited by 1

Not on the map. Why?

Abstract (arXiv)

In deep learning, the recently introduced state space models utilize HiPPO (High-order Polynomial Projection Operators) memory units to approximate continuous-time trajectories of input functions using ordinary differential equations (ODEs), and these techniques have shown empirical success in capturing long-range dependencies in long input sequences. However, the mathematical foundations of these ODEs, particularly the singular HiPPO-LegS (Legendre Scaled) ODE, and their corresponding numerical discretizations remain unsettled. In this work, we fill this gap by establishing that HiPPO-LegS ODE is well-posed despite its singularity, albeit without the freedom of arbitrary initial conditions. Further, we establish convergence of the associated numerical discretization schemes for Riemann integrable input functions.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 17 of its 34 references that are in the map. First posted on arXiv 11 Dec 2024.

Builds on

17 of 34 references.

References · 17

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  3. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  4. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  5. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,056
  6. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  7. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  8. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 605
  9. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 499
  10. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 328
  11. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  12. 2022Liquid Structural State-Space Modelscited by 136
5 more
  1. 2015A Simple Way to Initialize Recurrent Networks of Rectified Linear Unitscited by 130
  2. 2022How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projectionscited by 122
  3. 2024SSAMBA: Self-Supervised Audio Representation Learning with Mamba State Space Modelcited by 37
  4. 2021Parallelizing Legendre Memory Unit Trainingcited by 23
  5. 2024LMUFormer: Low Complexity Yet Powerful Spiking Model With Legendre Memory Unitscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Safety, Security, and Cognitive Risks in State-Space Models: A Systematic Threat Analysis with Spectral, Stateful, and Capacity Attackscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024HiPPO-Prophecy: State-Space Models can Provably Learn Dynamical Systems in Context
  2. 2026HiPPO Zoo: Explicit Memory Mechanisms for Interpretable State Space Models
  3. 2025UnHiPPO: Uncertainty-aware Initialization for State Space Models
  4. 2026Stability and Discretization Error of State Space Model Neural Operators
  5. 2024Towards a theory of learning dynamics in deep state space models
  6. 2024Deep Learning-based Approaches for State Space Models: A Selective Review
  7. 2026Latent Matters: Learning Deep State-Space Models
  8. 2025Hankel Singular Value Regularization for Highly Compressible State Space Models