# Easy attention: A simple attention mechanism for temporal predictions with transformers

arXiv: https://arxiv.org/abs/2308.12874 · HTML: https://mlsanity.com/paper/2308.12874/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 70 references that are in the map.
- First posted on arXiv 24 Aug 2023.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [Deep learning for universal linear embeddings of nonlinear dynamics](https://mlsanity.com/paper/1712.09707/) (2017)
- [An Introduction to Matrix Concentration Inequalities](https://mlsanity.com/paper/1501.01571/) (2015)
- [Choose a Transformer: Fourier or Galerkin](https://mlsanity.com/paper/2105.14995/) (2021)
- [Time2Vec: Learning a Vector Representation of Time](https://mlsanity.com/paper/1907.05321/) (2019)
- [Transformers for Modeling Physical Systems](https://mlsanity.com/paper/2010.03957/) (2020)
- [β-Variational autoencoders and transformers for reduced-order modelling of fluid flows](https://mlsanity.com/paper/2304.03571/) (2023)
- [Physics-Informed Probabilistic Learning of Linear Embeddings of Non-linear Dynamics With Guaranteed Stability](https://mlsanity.com/paper/1906.03663/) (2019)
- [Koopman Operator Dynamical Models: Learning, Analysis and Control](https://mlsanity.com/paper/2102.02522/) (2021)
- [Machine Learning Partners in Criminal Networks](https://mlsanity.com/paper/2209.03171/) (2022)

## Built on by (most cited first, 3 of 3)

- [Explainable AI: Learning from the Learners](https://mlsanity.com/paper/2601.05525/) (2026)
- [Transformer Learning of Chaotic Collective Dynamics in Many-Body Systems](https://mlsanity.com/paper/2601.19080/) (2026)
- [Equation-Free Digital Twins for Nonlinear Structural Dynamics](https://mlsanity.com/paper/2605.00950/) (2026)

## Abstract (arXiv)

To improve the robustness of transformer neural networks used for temporal-dynamics prediction of chaotic systems, we propose a novel attention mechanism called easy attention which we demonstrate in time-series reconstruction and prediction. While the standard self attention only makes use of the inner product of queries and keys, it is demonstrated that the keys, queries and softmax are not necessary for obtaining the attention score required to capture long-term dependencies in temporal sequences. Through the singular-value decomposition (SVD) on the softmax attention score, we further observe that self attention compresses the contributions from both queries and keys in the space spanned by the attention score. Therefore, our proposed easy-attention method directly treats the attention scores as learnable parameters. This approach produces excellent results when reconstructing and predicting the temporal dynamics of chaotic systems exhibiting more robustness and less complexity than self attention or the widely-used long short-term memory (LSTM) network. We show the improved performance of the easy-attention method in the Lorenz system, a turbulence shear flow and a model of a nuclear reactor.