mlsanityData to 7 Oct 2026

On the Benefits of Memory for Modeling Time-Dependent PDEs

arXiv 2409.02313 ↗First posted 3 Sep 2024cs.LG cs.AIcited by 30

Not on the map. Why?

Abstract (arXiv)

Data-driven techniques have emerged as a promising alternative to traditional numerical methods for solving PDEs. For time-dependent PDEs, many approaches are Markovian -- the evolution of the trained system only depends on the current state, and not the past states. In this work, we investigate the benefits of using memory for modeling time-dependent PDEs: that is, when past states are explicitly used to predict the future. Motivated by the Mori-Zwanzig theory of model reduction, we theoretically exhibit examples of simple (even linear) PDEs, in which a solution that uses memory is arbitrarily better than a Markovian solution. Additionally, we introduce Memory Neural Operator (MemNO), a neural operator architecture that combines recent state space models (specifically, S4) and Fourier Neural Operators (FNOs) to effectively model memory. We empirically demonstrate that when the PDEs are supplied in low resolution or contain observation noise at train and test time, MemNO significantly outperforms the baselines without memory -- with up to 6x reduction in test error. Furthermore, we show that this benefit is particularly pronounced when the PDE solutions have significant high-frequency Fourier modes (e.g., low-viscosity fluid dynamics) and we construct a challenging benchmark dataset consisting of such PDEs.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 38 of its 66 references that are in the map. First posted on arXiv 3 Sep 2024.

Builds on

38 of 66 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  7. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  9. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  10. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  11. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  12. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
26 more
  1. 2022FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operatorscited by 748
  2. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  3. 2019Deep Equilibrium Modelscited by 603
  4. 2021Physics-Informed Neural Operator for Learning Partial Differential Equationscited by 516
  5. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  6. 2022PDEBENCH: An Extensive Benchmark for Scientific Machine Learningcited by 396
  7. 2020Multipole Graph Neural Operator for Parametric Partial Differential Equationscited by 343
  8. 2021Choose a Transformer: Fourier or Galerkincited by 333
  9. 2023GNOT: A General Neural Operator Transformer for Operator Learningcited by 321
  10. 2023Geometry-Informed Neural Operator for Large-Scale 3D PDEscited by 293
  11. 2022Transformer for Partial Differential Equations' Operator Learningcited by 270
  12. 2022GraphCast: Learning skillful medium-range global weather forecastingcited by 261
  13. 2020Model Reduction and Neural Networks for Parametric PDEscited by 257
  14. 2021Factorized Fourier Neural Operatorscited by 237
  15. 2022U-NO: U-shaped Neural Operatorscited by 219
  16. 2023PDE-Refiner: Achieving Accurate Long Rollouts with Neural PDE Solverscited by 205
  17. 2022Towards Multi-spatiotemporal-scale Generalized PDE Modelingcited by 190
  18. 2024DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Trainingcited by 137
  19. 2023Scalable Transformer for PDE Surrogate Modelingcited by 123
  20. 2022How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projectionscited by 121
  21. 2023Multiple Physics Pretraining for Physical Surrogate Modelscited by 90
  22. 2024Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equationscited by 87
  23. 2020Neural Closure Models for Dynamical Systemscited by 22
  24. 2018Data-driven Discovery of Closure Modelscited by 19
  25. 2018Model Reduction with Memory and the Machine Learning of Dynamical Systemscited by 19
  26. 2023Deep Equilibrium Based Neural Operators for Steady-State PDEscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 15; 2026: 14.

30 papers.

Sort:

By area:

  1. 2025CodePDE: An Inference Framework for LLM-driven PDE Solver Generationcited by 37
  2. 2025Walrus: A Cross-Domain Foundation Model for Continuum Dynamicscited by 26
  3. 2024Mamba Neural Operator: Who Wins? Transformers vs. State-Space Models for PDEscited by 16
  4. 2025TI-DeepONet: Learnable Time Integration for Stable Long-Term Extrapolationcited by 14
  5. 2025Temporal Neural Operator for Modeling Time-Dependent Physical Phenomenacited by 8
  6. 2025Latent Mamba Operator for Partial Differential Equationscited by 8
  7. 2026AutoNumerics: An Autonomous, PDE-Agnostic Multi-Agent Pipeline for Scientific Computingcited by 7
  8. 2025Neural Emulator Superiority: When Machine Learning for PDEs Surpasses its Training Datacited by 6
  9. 2025Physics-Informed Time-Integrated DeepONet: Temporal Tangent Space Operator Learning for High-Accuracy Inferencecited by 5
  10. 2026Probabilistic Retrofitting of Learned Simulatorscited by 5
  11. 2025PDEfuncta: Spectrally-Aware Neural Representation for PDE Solution Modelingcited by 4
  12. 2025Predicting partially observable dynamical systems via diffusion models with a multiscale inference schemecited by 4
18 more
  1. 2026Memory-Conditioned Flow-Matching for Stable Autoregressive PDE Rolloutscited by 2
  2. 2025Differentiable Physics-Neural Models enable Learning of Non-Markovian Closures for Accelerated Coarse-Grained Physics Simulationscited by 1
  3. 2025Pre-Generating Multi-Difficulty PDE Data for Few-Shot Neural PDE Solverscited by 1
  4. 2026Flowers: A Warp Drive for Neural PDE Solverscited by 1
  5. 2025Physics-based machine learning for mantle convection simulationscited by 0
  6. 2025Merging Memory and Space: A State Space Neural Operatorcited by 0
  7. 2025Error Analysis of Generalized Langevin Equations with Approximated Memory Kernelscited by 0
  8. 2025ANCHOR: Error-Controlled Adaptive Numerical Correction for Neural Operator Time Marchingcited by 0
  9. 2026System-Anchored Knee Estimation for Low-Cost Context Window Selection in PDE Forecastingcited by 0
  10. 2026FlowForge: A Staged Local Rollout Engine for Flow-Field Predictioncited by 0
  11. 2026Posterior-First Neural PDE Simulation: Inferring Hidden Problem State from a Single Fieldcited by 0
  12. 2026Do Neural Operators Forget Geometry? The Forgetting Hypothesis in Deep Operator Learningcited by 0
  13. 2026How Much Memory Do We Need? Adaptive Memory Gate for Neural Operatorscited by 0
  14. 2026TF-SNO: Time-Frequency Gated Spectral Neural Operators for Learning Non-Stationary Partial Differential Equationscited by 0
  15. 2026No Free Lunch in Flow Surrogates under Time-Varying Boundary Conditions: A Two-Regime Studycited by 0
  16. 2026Why Does the Future Branch? Identifiable Closure Tests for Stochastic Physical World Modelscited by 0
  17. 2026DiffPDE: Masked Diffusion Language Models as PDE Solvercited by 0
  18. 2026Fractional Laplace Neural Operators: Exact Architectures, an Expressivity Frontier at Criticality, and Certified Stability for Memory-Driven Network Dynamicscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Autoregression-Free Neural Operators for Time-Dependent PDEs
  2. 2026Oscillatory State-Space Models as Inductive Biases for Physics-Informed Neural PDE Solvers
  3. 2025CFO: Learning Continuous-Time PDE Dynamics via Flow-Matched Neural Operators
  4. 2026A Constitutive Markov Physics-Informed Neural Operator (MPNO) for Autoregressive Stability in Transient Dynamics
  5. 2025Memory Determines Learning Direction: A Theory of Gradient-Based Optimization in State Space Models
  6. 2025Towards Reasoning for PDE Foundation Models: A Reward-Model-Driven Inference-Time-Scaling Algorithm
  7. 2024Diffusion models as probabilistic neural operators for recovering unobserved states of dynamical systems
  8. 2025ENMA: Tokenwise Autoregression for Generative Neural PDE Operators