# β-Variational autoencoders and transformers for reduced-order modelling of fluid flows

arXiv: https://arxiv.org/abs/2304.03571 · HTML: https://mlsanity.com/paper/2304.03571/

## Key facts

- Cited by 77 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 69 references that are in the map.
- First posted on arXiv 7 Apr 2023.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [Machine Learning for Fluid Mechanics](https://mlsanity.com/paper/1905.11075/) (2019)
- [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021)
- [Lookahead Optimizer: k steps forward, 1 step back](https://mlsanity.com/paper/1907.08610/) (2019)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [Time2Vec: Learning a Vector Representation of Time](https://mlsanity.com/paper/1907.05321/) (2019)
- [Enhancing Computational Fluid Dynamics with Machine Learning](https://mlsanity.com/paper/2110.02085/) (2021)
- [Transformers for Modeling Physical Systems](https://mlsanity.com/paper/2010.03957/) (2020)
- [A Survey of Deep Learning Techniques for Neural Machine Translation](https://mlsanity.com/paper/2002.07526/) (2020)
- [Towards extraction of orthogonal and parsimonious non-linear modes from turbulent flows](https://mlsanity.com/paper/2109.01514/) (2021)
- [Deep Neural Networks for Nonlinear Model Order Reduction of Unsteady Flows](https://mlsanity.com/paper/2007.00936/) (2020)
- [Image Captioning through Image Transformer](https://mlsanity.com/paper/2004.14231/) (2020)
- [Data-Driven POD-Galerkin Reduced Order Model for Turbulent Flows](https://mlsanity.com/paper/1907.09909/) (2019)
- [Predicting the temporal dynamics of turbulent channels through deep learning](https://mlsanity.com/paper/2203.00974/) (2022)

## Built on by (most cited first, 30 of 77)

- [Sparse identification of nonlinear dynamics and Koopman operators with Shallow Recurrent Decoder Networks](https://mlsanity.com/paper/2501.13329/) (2025)
- [Multi-scale Time-stepping of Partial Differential Equations with Transformers](https://mlsanity.com/paper/2311.02225/) (2023)
- [MORPH: PDE Foundation Models with Arbitrary Data Modality](https://mlsanity.com/paper/2509.21670/) (2025)
- [VENI, VINDy, VICI: a generative reduced-order modeling framework with uncertainty quantification](https://mlsanity.com/paper/2405.20905/) (2024)
- [Stochastic Latent Transformer: Efficient Modelling of Stochastically Forced Zonal Jets](https://mlsanity.com/paper/2310.16741/) (2023)
- [A conditional latent autoregressive recurrent model for generation and forecasting of beam dynamics in particle accelerators](https://mlsanity.com/paper/2403.13858/) (2024)
- [Diff-SPORT: Diffusion-based Sensor Placement Optimization and Reconstruction of Turbulent flows in urban environments](https://mlsanity.com/paper/2506.00214/) (2025)
- [Generative Adversarial Reduced Order Modelling](https://mlsanity.com/paper/2305.15881/) (2023)
- [Generative emulation of chaotic dynamics with coherent prior](https://mlsanity.com/paper/2504.14264/) (2025)
- [Generative prediction of flow fields around an obstacle using the diffusion model](https://mlsanity.com/paper/2407.00735/) (2024)
- [Time-inversion of spatiotemporal beam dynamics using uncertainty-aware latent evolution reversal](https://mlsanity.com/paper/2408.07847/) (2024)
- [Towards aerodynamic surrogate modeling based on β-variational autoencoders](https://mlsanity.com/paper/2408.04969/) (2024)
- [Reduced-order modeling of unsteady fluid flow using neural network ensembles](https://mlsanity.com/paper/2402.05372/) (2024)
- [Enhancing Graph U-Nets for Mesh-Agnostic Spatio-Temporal Flow Prediction](https://mlsanity.com/paper/2406.03789/) (2024)
- [Explainable AI: Learning from the Learners](https://mlsanity.com/paper/2601.05525/) (2026)
- [AeroJEPA: Learning Semantic Latent Representations for Scalable 3D Aerodynamic Field Modeling](https://mlsanity.com/paper/2605.05586/) (2026)
- [Data-driven Nonlinear Parametric Model Order Reduction Framework using Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2307.06816/) (2023)
- [Branched Latent Neural Maps](https://mlsanity.com/paper/2308.02599/) (2023)
- [Easy attention: A simple attention mechanism for temporal predictions with transformers](https://mlsanity.com/paper/2308.12874/) (2023)
- [Volume-Preserving Transformers for Learning Time Series Data with Structure](https://mlsanity.com/paper/2312.11166/) (2023)
- [Decoder Decomposition for the Analysis of the Latent Space of Nonlinear Autoencoders With Wind-Tunnel Experimental Data](https://mlsanity.com/paper/2404.19660/) (2024)
- [Decoding complexity: how machine learning is redefining scientific discovery](https://mlsanity.com/paper/2405.04161/) (2024)
- [Data-Augmented Predictive Deep Neural Network: Enhancing the extrapolation capabilities of non-intrusive surrogate models](https://mlsanity.com/paper/2410.13376/) (2024)
- [Model-Agnostic AI Framework with Explicit Time Integration for Long-Term Fluid Dynamics Prediction](https://mlsanity.com/paper/2412.05657/) (2024)
- [A Deep Learning approach for parametrized and time dependent Partial Differential Equations using Dimensionality Reduction and Neural ODEs](https://mlsanity.com/paper/2502.08683/) (2025)
- [Adaptive Diffusion Posterior Sampling for Data and Model Fusion of Complex Nonlinear Dynamical Systems](https://mlsanity.com/paper/2603.12635/) (2026)
- [A physics-informed neural network method for the approximation of slow invariant manifolds for the general class of stiff systems of ODEs](https://mlsanity.com/paper/2403.11591/) (2024)
- [Disentangled Latent Spaces for Reduced Order Models using Deterministic Autoencoders](https://mlsanity.com/paper/2502.14679/) (2025)
- [Regional climate projections using a deep-learning-based model-ranking and downscaling framework: Application to European climate zones](https://mlsanity.com/paper/2502.20132/) (2025)
- [Physics-informed Reduced Order Modeling of Time-dependent PDEs via Differentiable Solvers](https://mlsanity.com/paper/2505.14595/) (2025)

## Abstract (arXiv)

Variational autoencoder (VAE) architectures have the potential to develop reduced-order models (ROMs) for chaotic fluid flows. We propose a method for learning compact and near-orthogonal ROMs using a combination of a β-VAE and a transformer, tested on numerical data from a two-dimensional viscous flow in both periodic and chaotic regimes. The β-VAE is trained to learn a compact latent representation of the flow velocity, and the transformer is trained to predict the temporal dynamics in latent space. Using the β-VAE to learn disentangled representations in latent-space, we obtain a more interpretable flow model with features that resemble those observed in the proper orthogonal decomposition, but with a more efficient representation. Using Poincar'e maps, the results show that our method can capture the underlying dynamics of the flow outperforming other prediction models. The proposed method has potential applications in other fields such as weather forecasting, structural dynamics or biomedical engineering.