mlsanityData to 7 Oct 2026

How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projections

arXiv 2206.12037 ↗First posted 24 Jun 2022cs.LGcited by 121

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

Linear time-invariant state space models (SSM) are a classical model from engineering and statistics, that have recently been shown to be very promising in machine learning through the Structured State Space sequence model (S4). A core component of S4 involves initializing the SSM state matrix to a particular matrix called a HiPPO matrix, which was empirically important for S4's ability to handle long sequences. However, the specific matrix that S4 uses was actually derived in previous work for a particular time-varying dynamical system, and the use of this matrix as a time-invariant SSM had no known mathematical interpretation. Consequently, the theoretical mechanism by which S4 models long-range dependencies actually remains unexplained. We derive a more general and intuitive formulation of the HiPPO framework, which provides a simple mathematical interpretation of S4 as a decomposition onto exponentially-warped Legendre polynomials, explaining its ability to capture long dependencies. Our generalization introduces a theoretically rich class of SSMs that also lets us derive more intuitive S4 variants for other bases such as the Fourier basis, and explains other aspects of training S4, such as how to initialize the important timescale parameter. These insights improve S4's performance to 86% on the Long Range Arena benchmark, with 96% on the most difficult Path-X task.

Key facts. Cited by 121 papers in the map (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 8 of its 20 references that are in the map. First posted on arXiv 24 Jun 2022.

Builds on

8 of 20 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Diagonal State Spaces are as Effective as Structured State Spaces27 Mar
  2. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 6

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  2. 2016Layer Normalizationcited by 5,579
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  4. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  5. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  6. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 12; 2024: 45; 2025: 42; 2026: 16.

121 papers.

Sort:

By area:

  1. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  2. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  3. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  4. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  5. 2022Diffusion-based Time Series Imputation and Forecasting with Structured State Space Modelscited by 195
  6. 2024PointMamba: A Simple State Space Model for Point Cloud Analysiscited by 166
  7. 2024PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognitioncited by 141
  8. 2022Liquid Structural State-Space Modelscited by 134
  9. 2022What Makes Convolutional Models Great on Long Sequence Modeling?cited by 86
  10. 2024User-LLM: Efficient LLM Contextualization with User Embeddingscited by 74
  11. 2024DiM: Diffusion Mamba for Efficient High-Resolution Image Synthesiscited by 72
  12. 2023Scallop: A Language for Neurosymbolic Programmingcited by 63
38 more
  1. 2023Simple Hardware-Efficient Long Convolutions for Sequence Modelingcited by 62
  2. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  3. 2024MambaByte: Token-free Selective State Space Modelcited by 58
  4. 2024Gamba: Marry Gaussian Splatting with Mamba for single view 3D reconstructioncited by 56
  5. 2022S4ND: Modeling Images and Videos as Multidimensional Signals Using State Spacescited by 41
  6. 2024Visual Mamba: A Survey and New Outlookscited by 41
  7. 2024Mamba-R: Vision Mamba ALSO Needs Registerscited by 41
  8. 2023State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memorycited by 38
  9. 2024Oscillatory State-Space Modelscited by 35
  10. 2023FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Corescited by 32
  11. 2023Facing Off World Model Backbones: RNNs, Transformers, and S4cited by 31
  12. 2024On the Benefits of Memory for Modeling Time-Dependent PDEscited by 30
  13. 2024The Rise of Diffusion Models in Time-Series Forecastingcited by 25
  14. 2024Autoregressive Pretraining with Mamba in Visioncited by 21
  15. 2023Robustifying State-space Models for Long Sequences via Approximate Diagonalizationcited by 18
  16. 2024State-Free Inference of State-Space Models: The Transfer Function Approachcited by 18
  17. 2024Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothingcited by 17
  18. 2023Neural Oscillators are Universalcited by 16
  19. 2024Text-controlled Motion Mamba: Text-Instructed Temporal Grounding of Human Motioncited by 16
  20. 2024Tuning Frequency Bias of State Space Modelscited by 16
  21. 2024MMR-Mamba: Multi-Modal MRI Reconstruction with Mamba and Spatial-Frequency Information Fusioncited by 15
  22. 2023Advancing Regular Language Reasoning in Linear Recurrent Neural Networkscited by 13
  23. 2025Comba: Improving Bilinear RNNs with Closed-loop Controlcited by 12
  24. 2023Recurrent Distance Filtering for Graph Representation Learningcited by 11
  25. 2024From Generalization Analysis to Optimization Designs for State Space Modelscited by 11
  26. 2024HOPE for a Robust Parameterization of Long-memory State Space Modelscited by 11
  27. 2025Video Latent Flow Matching: Optimal Polynomial Projections for Video Interpolation and Extrapolationcited by 11
  28. 2024Linear Transformers with Learnable Kernel Functions are Better In-Context Modelscited by 10
  29. 2024Time-SSM: Simplifying and Unifying State Space Models for Time Series Forecastingcited by 9
  30. 2024Adventurer: Optimizing Vision Mamba Architecture Designs for Efficiencycited by 9
  31. 2024PTQ4VM: Post-Training Quantization for Visual Mambacited by 9
  32. 2025Skip Mamba Diffusion for Monocular 3D Semantic Scene Completioncited by 9
  33. 2025Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mambacited by 9
  34. 2024Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Modelscited by 8
  35. 2025End-to-end workflow for machine learning-based qubit readout with QICK and hls4mlcited by 8
  36. 2025Generalizable, real-time neural decoding with hybrid state-space modelscited by 8
  37. 2025FlowState: Sampling-Rate-Equivariant Time-Series Forecastingcited by 7
  38. 2025TransMamba: Fast Universal Architecture Adaption from Transformers to Mambacited by 6

50 most cited of 121 All 121 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024State Space Models as Foundation Models: A Control Theoretic Overview
  2. 2024Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Models
  3. 2024Deep Learning-based Approaches for State Space Models: A Selective Review
  4. 2026A Simple State Space Model Excels at Multivariate Time Series Classification
  5. 2022Hidden Parameter Recurrent State Space Models For Changing Dynamics Scenarios
  6. 2023Effectively Modeling Time Series with Simple Discrete State Spaces
  7. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges
  8. 2024Incorporating Exponential Smoothing into MLP: A Simple but Effective Sequence Model