# Training neural operators to preserve invariant measures of chaotic attractors

arXiv: https://arxiv.org/abs/2306.01187 · HTML: https://mlsanity.com/paper/2306.01187/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 58 references that are in the map.
- First posted on arXiv 1 Jun 2023.

## Builds on (most cited first, 30 of 32)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [PDEBENCH: An Extensive Benchmark for Scientific Machine Learning](https://mlsanity.com/paper/2210.07182/) (2022)
- [Message Passing Neural PDE Solvers](https://mlsanity.com/paper/2202.03376/) (2022)
- [Transformer for Partial Differential Equations' Operator Learning](https://mlsanity.com/paper/2205.13671/) (2022)
- [Learning Local Equivariant Representations for Large-Scale Atomistic Dynamics](https://mlsanity.com/paper/2204.05249/) (2022)
- [Machine Learning Force Fields](https://mlsanity.com/paper/2010.07067/) (2020)
- [Towards Multi-spatiotemporal-scale Generalized PDE Modeling](https://mlsanity.com/paper/2209.15616/) (2022)
- [Computational Optimal Transport: Complexity by Accelerated Gradient Descent Is Better Than by Sinkhorn's Algorithm](https://mlsanity.com/paper/1802.04367/) (2018)
- [Multiscale Simulations of Complex Systems by Learning their Effective Dynamics](https://mlsanity.com/paper/2006.13431/) (2020)
- [Lie Point Symmetry Data Augmentation for Neural PDE Solvers](https://mlsanity.com/paper/2202.07643/) (2022)
- [On the difficulty of learning chaotic dynamics with RNNs](https://mlsanity.com/paper/2110.07238/) (2021)
- [Self-Supervised Visual Representation Learning from Hierarchical Grouping](https://mlsanity.com/paper/2012.03044/) (2020)
- [Generative Adversarial Neural Operators](https://mlsanity.com/paper/2205.03017/) (2022)
- [Generalizing to New Physical Systems via Context-Informed Dynamics Model](https://mlsanity.com/paper/2202.01889/) (2022)
- [Meta-Learning Dynamics Forecasting Using Task Inference](https://mlsanity.com/paper/2102.10271/) (2021)
- [Debiased Sinkhorn barycenters](https://mlsanity.com/paper/2006.02575/) (2020)
- [Auto-differentiable Ensemble Kalman Filters](https://mlsanity.com/paper/2107.07687/) (2021)
- [Scalable Optimal Transport Methods in Machine Learning: A Contemporary Survey](https://mlsanity.com/paper/2305.05080/) (2023)
- [Extracting Interpretable Physical Parameters from Spatiotemporal Systems using Unsupervised Learning](https://mlsanity.com/paper/1907.06011/) (2019)
- [Discovering Conservation Laws using Optimal Transport and Manifold Learning](https://mlsanity.com/paper/2208.14995/) (2022)
- [Semantic-Aware Generation for Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/2111.13163/) (2021)

## Built on by (most cited first, 30 of 46)

- [Probabilistic Forecasting with Stochastic Interpolants and F"ollmer Processes](https://mlsanity.com/paper/2403.13724/) (2024)
- [Out-of-Domain Generalization in Dynamical Systems Reconstruction](https://mlsanity.com/paper/2402.18377/) (2024)
- [DySLIM: Dynamics Stable Learning by Invariant Measure for Chaotic Systems](https://mlsanity.com/paper/2402.04467/) (2024)
- [Non-stationary Diffusion For Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2505.04278/) (2025)
- [Thermalizer: Stable autoregressive neural emulation of spatiotemporal chaos](https://mlsanity.com/paper/2503.18731/) (2025)
- [True Zero-Shot Inference of Dynamical Systems Preserving Long-Term Statistics](https://mlsanity.com/paper/2505.13192/) (2025)
- [Learning Interpretable Hierarchical Dynamical Systems Models from Time Series Data](https://mlsanity.com/paper/2410.04814/) (2024)
- [When are dynamical systems learned from time series data statistically accurate?](https://mlsanity.com/paper/2411.06311/) (2024)
- [Binned Spectral Power Loss for Improved Prediction of Chaotic Systems](https://mlsanity.com/paper/2502.00472/) (2025)
- [Hierarchical Implicit Neural Emulators](https://mlsanity.com/paper/2506.04528/) (2025)
- [A scalable generative model for dynamical system reconstruction from neuroimaging data](https://mlsanity.com/paper/2411.02949/) (2024)
- [Project and Generate: Divergence-Free Neural Operators for Incompressible Flows](https://mlsanity.com/paper/2603.24500/) (2026)
- [PICL: Physics Informed Contrastive Learning for Partial Differential Equations](https://mlsanity.com/paper/2401.16327/) (2024)
- [A probabilistic framework for learning non-intrusive corrections to long-time climate simulations from short-time training data](https://mlsanity.com/paper/2408.02688/) (2024)
- [Invariant Measures in Time-Delay Coordinates for Unique Dynamical System Identification](https://mlsanity.com/paper/2412.00589/) (2024)
- [Position: A Dynamical Systems Perspective is Needed to Advance Time Series Modeling](https://mlsanity.com/paper/2602.16864/) (2026)
- [Embed and Emulate: Contrastive representations for simulation-based inference](https://mlsanity.com/paper/2409.18402/) (2024)
- [INC: An Indirect Neural Corrector for Auto-Regressive Hybrid PDE Solvers](https://mlsanity.com/paper/2511.12764/) (2025)
- [Learning Lagrangian Interaction Dynamics with Sampling-Based Model Order Reduction](https://mlsanity.com/paper/2407.03925/) (2024)
- [Nested Diffusion Models Using Hierarchical Latent Priors](https://mlsanity.com/paper/2412.05984/) (2024)
- [Geometric Neural Operators via Lie Group-Constrained Latent Dynamics](https://mlsanity.com/paper/2602.16209/) (2026)
- [Long-time accuracy of ensemble Kalman filters for chaotic and machine-learned dynamical systems](https://mlsanity.com/paper/2412.14318/) (2024)
- [Invariant Measures for Data-Driven Dynamical System Identification: Analysis and Application](https://mlsanity.com/paper/2502.05204/) (2025)
- [ChaosNexus: A Foundation Model for ODE-based Chaotic System Forecasting with Hierarchical Multi-scale Awareness](https://mlsanity.com/paper/2509.21802/) (2025)
- [Towards Infinitely Long Neural Simulations: Self-Refining Neural Surrogate Models for Dynamical Systems](https://mlsanity.com/paper/2603.17750/) (2026)
- [The Dynamic-Probabilistic Consistency Gap in Chaotic Surrogate Modeling](https://mlsanity.com/paper/2605.31547/) (2026)
- [First-Order Trajectory Matching: Fast Ensemble Predictions of Chaotic, Turbulent, Stochastic Systems](https://mlsanity.com/paper/2606.11138/) (2026)
- [HERO: History-Enriched Rollout Training for Long-Horizon Autoregressive Neural Operators](https://mlsanity.com/paper/2607.29135/) (2026)
- [WaveCastNet: Rapid Wavefield Forecasting for Earthquake Early Warning via Deep Sequence to Sequence Learning](https://mlsanity.com/paper/2405.20516/) (2024)
- [Ellipsoidal Time Series Forecasting](https://mlsanity.com/paper/2505.17370/) (2025)

## Abstract (arXiv)

Chaotic systems make long-horizon forecasts difficult because small perturbations in initial conditions cause trajectories to diverge at an exponential rate. In this setting, neural operators trained to minimize squared error losses, while capable of accurate short-term forecasts, often fail to reproduce statistical or structural properties of the dynamics over longer time horizons and can yield degenerate results. In this paper, we propose an alternative framework designed to preserve invariant measures of chaotic attractors that characterize the time-invariant statistical properties of the dynamics. Specifically, in the multi-environment setting (where each sample trajectory is governed by slightly different dynamics), we consider two novel approaches to training with noisy data. First, we propose a loss based on the optimal transport distance between the observed dynamics and the neural operator outputs. This approach requires expert knowledge of the underlying physics to determine what statistical features should be included in the optimal transport loss. Second, we show that a contrastive learning framework, which does not require any specialized prior knowledge, can preserve statistical properties of the dynamics nearly as well as the optimal transport approach. On a variety of chaotic systems, our method is shown empirically to preserve invariant measures of chaotic attractors.