# On the Benefits of Memory for Modeling Time-Dependent PDEs

arXiv: https://arxiv.org/abs/2409.02313 · HTML: https://mlsanity.com/paper/2409.02313/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 66 references that are in the map.
- First posted on arXiv 3 Sep 2024.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [Physics-Informed Neural Operator for Learning Partial Differential Equations](https://mlsanity.com/paper/2111.03794/) (2021)
- [PDEBENCH: An Extensive Benchmark for Scientific Machine Learning](https://mlsanity.com/paper/2210.07182/) (2022)
- [Resurrecting Recurrent Neural Networks for Long Sequences](https://mlsanity.com/paper/2303.06349/) (2023)
- [Multipole Graph Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2006.09535/) (2020)
- [Choose a Transformer: Fourier or Galerkin](https://mlsanity.com/paper/2105.14995/) (2021)
- [GNOT: A General Neural Operator Transformer for Operator Learning](https://mlsanity.com/paper/2302.14376/) (2023)
- [Geometry-Informed Neural Operator for Large-Scale 3D PDEs](https://mlsanity.com/paper/2309.00583/) (2023)
- [Transformer for Partial Differential Equations' Operator Learning](https://mlsanity.com/paper/2205.13671/) (2022)
- [GraphCast: Learning skillful medium-range global weather forecasting](https://mlsanity.com/paper/2212.12794/) (2022)
- [Model Reduction and Neural Networks for Parametric PDEs](https://mlsanity.com/paper/2005.03180/) (2020)
- [Factorized Fourier Neural Operators](https://mlsanity.com/paper/2111.13802/) (2021)
- [U-NO: U-shaped Neural Operators](https://mlsanity.com/paper/2204.11127/) (2022)
- [PDE-Refiner: Achieving Accurate Long Rollouts with Neural PDE Solvers](https://mlsanity.com/paper/2308.05732/) (2023)
- [Towards Multi-spatiotemporal-scale Generalized PDE Modeling](https://mlsanity.com/paper/2209.15616/) (2022)
- [DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training](https://mlsanity.com/paper/2403.03542/) (2024)

## Built on by (most cited first, 30 of 31)

- [CodePDE: An Inference Framework for LLM-driven PDE Solver Generation](https://mlsanity.com/paper/2505.08783/) (2025)
- [Walrus: A Cross-Domain Foundation Model for Continuum Dynamics](https://mlsanity.com/paper/2511.15684/) (2025)
- [Mamba Neural Operator: Who Wins? Transformers vs. State-Space Models for PDEs](https://mlsanity.com/paper/2410.02113/) (2024)
- [TI-DeepONet: Learnable Time Integration for Stable Long-Term Extrapolation](https://mlsanity.com/paper/2505.17341/) (2025)
- [Latent Mamba Operator for Partial Differential Equations](https://mlsanity.com/paper/2505.19105/) (2025)
- [Temporal Neural Operator for Modeling Time-Dependent Physical Phenomena](https://mlsanity.com/paper/2504.20249/) (2025)
- [AutoNumerics: An Autonomous, PDE-Agnostic Multi-Agent Pipeline for Scientific Computing](https://mlsanity.com/paper/2602.17607/) (2026)
- [Probabilistic Retrofitting of Learned Simulators](https://mlsanity.com/paper/2603.01949/) (2026)
- [Neural Emulator Superiority: When Machine Learning for PDEs Surpasses its Training Data](https://mlsanity.com/paper/2510.23111/) (2025)
- [Physics-Informed Time-Integrated DeepONet: Temporal Tangent Space Operator Learning for High-Accuracy Inference](https://mlsanity.com/paper/2508.05190/) (2025)
- [Predicting partially observable dynamical systems via diffusion models with a multiscale inference scheme](https://mlsanity.com/paper/2511.19390/) (2025)
- [PDEfuncta: Spectrally-Aware Neural Representation for PDE Solution Modeling](https://mlsanity.com/paper/2506.12790/) (2025)
- [Merging Memory and Space: A State Space Neural Operator](https://mlsanity.com/paper/2507.23428/) (2025)
- [Memory-Conditioned Flow-Matching for Stable Autoregressive PDE Rollouts](https://mlsanity.com/paper/2602.06689/) (2026)
- [Flowers: A Warp Drive for Neural PDE Solvers](https://mlsanity.com/paper/2603.04430/) (2026)
- [Differentiable Physics-Neural Models enable Learning of Non-Markovian Closures for Accelerated Coarse-Grained Physics Simulations](https://mlsanity.com/paper/2511.21369/) (2025)
- [Pre-Generating Multi-Difficulty PDE Data for Few-Shot Neural PDE Solvers](https://mlsanity.com/paper/2512.00564/) (2025)
- [System-Anchored Knee Estimation for Low-Cost Context Window Selection in PDE Forecasting](https://mlsanity.com/paper/2603.25025/) (2026)
- [How Much Memory Do We Need? Adaptive Memory Gate for Neural Operators](https://mlsanity.com/paper/2606.13443/) (2026)
- [Physics-based machine learning for mantle convection simulations](https://mlsanity.com/paper/2505.16041/) (2025)
- [Error Analysis of Generalized Langevin Equations with Approximated Memory Kernels](https://mlsanity.com/paper/2512.10256/) (2025)
- [ANCHOR: Error-Controlled Adaptive Numerical Correction for Neural Operator Time Marching](https://mlsanity.com/paper/2512.19643/) (2025)
- [FlowForge: A Staged Local Rollout Engine for Flow-Field Prediction](https://mlsanity.com/paper/2604.18953/) (2026)
- [Posterior-First Neural PDE Simulation: Inferring Hidden Problem State from a Single Field](https://mlsanity.com/paper/2605.03247/) (2026)
- [Do Neural Operators Forget Geometry? The Forgetting Hypothesis in Deep Operator Learning](https://mlsanity.com/paper/2605.05862/) (2026)
- [TF-SNO: Time-Frequency Gated Spectral Neural Operators for Learning Non-Stationary Partial Differential Equations](https://mlsanity.com/paper/2606.21189/) (2026)
- [No Free Lunch in Flow Surrogates under Time-Varying Boundary Conditions: A Two-Regime Study](https://mlsanity.com/paper/2607.23667/) (2026)
- [Why Does the Future Branch? Identifiable Closure Tests for Stochastic Physical World Models](https://mlsanity.com/paper/2608.00591/) (2026)
- [DiffPDE: Masked Diffusion Language Models as PDE Solver](https://mlsanity.com/paper/2608.30532/) (2026)
- [Fractional Laplace Neural Operators: Exact Architectures, an Expressivity Frontier at Criticality, and Certified Stability for Memory-Driven Network Dynamics](https://mlsanity.com/paper/2610.00515/) (2026)

## Abstract (arXiv)

Data-driven techniques have emerged as a promising alternative to traditional numerical methods for solving PDEs. For time-dependent PDEs, many approaches are Markovian -- the evolution of the trained system only depends on the current state, and not the past states. In this work, we investigate the benefits of using memory for modeling time-dependent PDEs: that is, when past states are explicitly used to predict the future. Motivated by the Mori-Zwanzig theory of model reduction, we theoretically exhibit examples of simple (even linear) PDEs, in which a solution that uses memory is arbitrarily better than a Markovian solution. Additionally, we introduce Memory Neural Operator (MemNO), a neural operator architecture that combines recent state space models (specifically, S4) and Fourier Neural Operators (FNOs) to effectively model memory. We empirically demonstrate that when the PDEs are supplied in low resolution or contain observation noise at train and test time, MemNO significantly outperforms the baselines without memory -- with up to 6x reduction in test error. Furthermore, we show that this benefit is particularly pronounced when the PDE solutions have significant high-frequency Fourier modes (e.g., low-viscosity fluid dynamics) and we construct a challenging benchmark dataset consisting of such PDEs.