mlsanityData to 7 Oct 2026

HiPPO: Recurrent Memory with Optimal Polynomial Projections

arXiv 2008.07669 ↗First posted 17 Aug 2020cs.LG stat.MLcited by 600

Not on the map. Why?

Cited by 223 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A central problem in learning from sequential data is representing cumulative history in an incremental fashion as more data is processed. We introduce a general framework (HiPPO) for the online compression of continuous signals and discrete time series by projection onto polynomial bases. Given a measure that specifies the importance of each time step in the past, HiPPO produces an optimal solution to a natural online function approximation problem. As special cases, our framework yields a short derivation of the recent Legendre Memory Unit (LMU) from first principles, and generalizes the ubiquitous gating mechanism of recurrent neural networks such as GRUs. This formal framework yields a new memory update mechanism (HiPPO-LegS) that scales through time to remember all history, avoiding priors on the timescale. HiPPO-LegS enjoys the theoretical benefits of timescale robustness, fast updates, and bounded gradients. By incorporating the memory dynamics into recurrent neural networks, HiPPO RNNs can empirically capture complex temporal dependencies. On the benchmark permuted MNIST dataset, HiPPO-LegS sets a new state-of-the-art accuracy of 98.3%. Finally, on a novel trajectory classification task testing robustness to out-of-distribution timescales and missing data, HiPPO-LegS outperforms RNN and neural ODE baselines by 25-40% accuracy.

Key facts. Cited by 600 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Aug 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 34; 2023: 55; 2024: 215; 2025: 168; 2026: 117.

600 papers.

Sort:

By area:

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  4. 2024VMamba: Visual State Space Modelcited by 955
  5. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  6. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  7. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  8. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  9. 2024U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentationcited by 412
  10. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  11. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  12. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
38 more
  1. 2024MambaIR: A Simple Baseline for Image Restoration with State-Space Modelcited by 349
  2. 2024VM-UNet: Vision Mamba UNet for Medical Image Segmentationcited by 286
  3. 2022Long Range Language Modeling via Gated State Spacescited by 265
  4. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  5. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  6. 2021Multiwavelet-based Operator Learning for Differential Equationscited by 229
  7. 2023PDE-Refiner: Achieving Accurate Long Rollouts with Neural PDE Solverscited by 205
  8. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  9. 2022Diffusion-based Time Series Imputation and Forecasting with Structured State Space Modelscited by 195
  10. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  11. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  12. 2024PointMamba: A Simple State Space Model for Point Cloud Analysiscited by 166
  13. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 155
  14. 2022FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecastingcited by 147
  15. 2022Liquid Structural State-Space Modelscited by 134
  16. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 128
  17. 2022How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projectionscited by 121
  18. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
  19. 2021CKConv: Continuous Kernel Convolution For Sequential Datacited by 117
  20. 2024Is Mamba Effective for Time Series Forecasting?cited by 110
  21. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 109
  22. 2022Long Movie Clip Classification with State-Space Video Modelscited by 106
  23. 2023Selective Structured State-Spaces for Long-Form Video Understandingcited by 105
  24. 2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 100
  25. 2024VL-Mamba: Exploring State Space Models for Multimodal Learningcited by 93
  26. 2022What Makes Convolutional Models Great on Long Sequence Modeling?cited by 86
  27. 2024Graph Mamba: Towards Learning on Graphs with State Space Modelscited by 84
  28. 2024SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate Time seriescited by 81
  29. 2024RS-Mamba for Large Remote Sensing Image Dense Predictioncited by 80
  30. 2023Diffusion Models Without Attentioncited by 75
  31. 2024State Space Models for Event Camerascited by 75
  32. 2024Theoretical Foundations of Deep Selective State-Space Modelscited by 73
  33. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 71
  34. 2024Point Mamba: A Novel Point Cloud Backbone Based on State Space Model with Octree-Based Ordering Strategycited by 71
  35. 2021FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizescited by 66
  36. 2021Closed-form Continuous-time Neural Modelscited by 65
  37. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  38. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 63

50 most cited of 600 All 600 (JSON)