# Variational autoencoders with latent high-dimensional steady geometric flows for dynamics

arXiv: https://arxiv.org/abs/2410.10137 · HTML: https://mlsanity.com/paper/2410.10137/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 56 references that are in the map.
- First posted on arXiv 14 Oct 2024.

## Builds on (most cited first, 28 of 28)

- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Learning the solution operator of parametric partial differential equations with physics-informed DeepOnets](https://mlsanity.com/paper/2103.10974/) (2021)
- [Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations](https://mlsanity.com/paper/1711.10561/) (2017)
- [Hyperspherical Variational Auto-Encoders](https://mlsanity.com/paper/1804.00891/) (2018)
- [Robustness via curvature regularization, and vice versa](https://mlsanity.com/paper/1811.09716/) (2018)
- [Respecting causality is all you need for training physics-informed neural networks](https://mlsanity.com/paper/2203.07404/) (2022)
- [Understanding and mitigating gradient pathologies in physics-informed neural networks](https://mlsanity.com/paper/2001.04536/) (2020)
- [Tackling the Curse of Dimensionality with Physics-Informed Neural Networks](https://mlsanity.com/paper/2307.12306/) (2023)
- [Diffusion Normalizing Flow](https://mlsanity.com/paper/2110.07579/) (2021)
- [Normalizing Flows on Riemannian Manifolds](https://mlsanity.com/paper/1611.02304/) (2016)
- [Variational Autoencoder with Learned Latent Structure](https://mlsanity.com/paper/2006.10597/) (2020)
- [Integrating Neural Operators with Diffusion Models Improves Spectral Representation in Turbulence Modeling](https://mlsanity.com/paper/2409.08477/) (2024)
- [RiemannONets: Interpretable Neural Operators for Riemann Problems](https://mlsanity.com/paper/2401.08886/) (2024)
- [State-space models are accurate and efficient neural operators for dynamical systems](https://mlsanity.com/paper/2409.03231/) (2024)
- [A Geometric Perspective on Variational Autoencoders](https://mlsanity.com/paper/2209.07370/) (2022)
- [Ultrahyperbolic Representation Learning](https://mlsanity.com/paper/2007.00211/) (2020)
- [Real-time Inference and Extrapolation via a Diffusion-inspired Temporal Transformer Operator (DiTTO)](https://mlsanity.com/paper/2307.09072/) (2023)
- [Adversarial Autoencoders with Constant-Curvature Latent Manifolds](https://mlsanity.com/paper/1812.04314/) (2018)
- [Ensemble learning for Physics Informed Neural Networks: a Gradient Boosting approach](https://mlsanity.com/paper/2302.13143/) (2023)
- [Representing Deep Neural Networks Latent Space Geometries with Graphs](https://mlsanity.com/paper/2011.07343/) (2020)
- [FastVPINNs: Tensor-Driven Acceleration of VPINNs for Complex Geometries](https://mlsanity.com/paper/2404.12063/) (2024)
- [Repulsive Curves](https://mlsanity.com/paper/2006.07859/) (2020)
- [On Manifold Hypothesis: Hypersurface Submanifold Embedding Using Osculating Hyperspheres](https://mlsanity.com/paper/2202.01619/) (2022)
- [Manifold Adversarial Learning](https://mlsanity.com/paper/1807.05832/) (2018)
- [Manifold Hypothesis in Data Analysis: Double Geometrically-Probabilistic Approach to Manifold Dimension Estimation](https://mlsanity.com/paper/2107.03903/) (2021)

## Abstract (arXiv)

We develop Riemannian approaches to variational autoencoders (VAEs) for PDE-type ambient data with regularizing geometric latent dynamics, which we refer to as VAE-DLM, or VAEs with dynamical latent manifolds. We redevelop the VAE framework such that manifold geometries, subject to our geometric flow, embedded in Euclidean space are learned in the intermediary latent space developed by encoders and decoders. By tailoring the geometric flow in which the latent space evolves, we induce latent geometric properties of our choosing, which are reflected in empirical performance. We reformulate the traditional evidence lower bound (ELBO) loss with a considerate choice of prior. We develop a linear geometric flow with a steady-state regularizing term. This flow requires only automatic differentiation of one time derivative, and can be solved in moderately high dimensions in a physics-informed approach, allowing more expressive latent representations. We discuss how this flow can be formulated as a gradient flow, and maintains entropy away from metric singularity. This, along with an eigenvalue penalization condition, helps ensure the manifold is sufficiently large in measure, nondegenerate, and a canonical geometry, which contribute to a robust representation. Our methods focus on the modified multi-layer perceptron architecture with tanh activations for the manifold encoder-decoder. We demonstrate, on our datasets of interest, our methods perform at least as well as the traditional VAE, and oftentimes better. Our methods can outperform this and a VAE endowed with our proposed architecture, frequently reducing out-of-distribution (OOD) error between 15% to 35% on select datasets. We highlight our method on ambient PDEs whose solutions maintain minimal variation in late times. We provide empirical justification towards how we can improve robust learning for external dynamics with VAEs.