# Volume-Preserving Transformers for Learning Time Series Data with Structure

arXiv: https://arxiv.org/abs/2312.11166 · HTML: https://mlsanity.com/paper/2312.11166/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 40 references that are in the map.
- First posted on arXiv 18 Dec 2023.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Hamiltonian Neural Networks](https://mlsanity.com/paper/1906.01563/) (2019)
- [A comprehensive deep learning-based approach to reduced order modeling of nonlinear time-dependent parametrized PDEs](https://mlsanity.com/paper/2001.04001/) (2020)
- [β-Variational autoencoders and transformers for reduced-order modelling of fluid flows](https://mlsanity.com/paper/2304.03571/) (2023)
- [Transformer Meets Boundary Value Inverse Problems](https://mlsanity.com/paper/2209.14977/) (2022)
- [Effective Extensible Programming: Unleashing Julia on GPUs](https://mlsanity.com/paper/1712.03112/) (2017)
- [Discovering State Variables Hidden in Experimental Data](https://mlsanity.com/paper/2112.10755/) (2021)
- [Locally-symplectic neural networks for learning volume-preserving dynamics](https://mlsanity.com/paper/2109.09151/) (2021)
- [Symplectic Autoencoders for Model Reduction of Hamiltonian Systems](https://mlsanity.com/paper/2312.10004/) (2023)
- [LQResNet: A Deep Neural Network Architecture for Learning Dynamic Processes](https://mlsanity.com/paper/2103.02249/) (2021)

## Built on by (most cited first, 3 of 3)

- [PHAST: Port-Hamiltonian Architecture for Structured Temporal Dynamics Forecasting](https://mlsanity.com/paper/2602.17998/) (2026)
- [Can Transformers overcome the lack of data in the simulation of history-dependent flows?](https://mlsanity.com/paper/2512.16305/) (2025)
- [An End-to-End PyTorch Interface for Differentiable PDE Solvers: A RANS Model-Correction Study](https://mlsanity.com/paper/2605.28858/) (2026)

## Abstract (arXiv)

Two of the many trends in neural network research of the past few years have been (i) the learning of dynamical systems, especially with recurrent neural networks such as long short-term memory networks (LSTMs) and (ii) the introduction of transformer neural networks for natural language processing (NLP) tasks. While some work has been performed on the intersection of these two trends, those efforts were largely limited to using the vanilla transformer directly without adjusting its architecture for the setting of a physical system. In this work we develop a transformer-inspired neural network and use it to learn a dynamical system. We (for the first time) change the activation function of the attention layer to imbue the transformer with structure-preserving properties to improve long-term stability. This is shown to be of great advantage when applying the neural network to learning the trajectory of a rigid body.