# Euclidean Fast Attention -- Machine Learning Global Atomic Representations at Linear Cost

arXiv: https://arxiv.org/abs/2412.08541 · HTML: https://mlsanity.com/paper/2412.08541/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 86 references that are in the map.
- First posted on arXiv 11 Dec 2024.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [On the Bottleneck of Graph Neural Networks and its Practical Implications](https://mlsanity.com/paper/2006.05205/) (2020)
- [3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data](https://mlsanity.com/paper/1807.02547/) (2018)
- [Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs](https://mlsanity.com/paper/2206.11990/) (2022)
- [Learning Local Equivariant Representations for Large-Scale Atomistic Dynamics](https://mlsanity.com/paper/2204.05249/) (2022)
- [Machine Learning Force Fields](https://mlsanity.com/paper/2010.07067/) (2020)
- [SpookyNet: Learning Force Fields with Electronic Degrees of Freedom and Nonlocal Effects](https://mlsanity.com/paper/2105.00304/) (2021)
- [Forces are not Enough: Benchmark and Critical Evaluation for Machine Learning Force Fields with Molecular Simulations](https://mlsanity.com/paper/2210.07237/) (2022)
- [On the Expressive Power of Geometric Graph Neural Networks](https://mlsanity.com/paper/2301.09308/) (2023)
- [SPICE, A Dataset of Drug-like Molecules and Peptides for Training Machine Learning Potentials](https://mlsanity.com/paper/2209.10702/) (2022)
- [SE(3)-equivariant prediction of molecular wavefunctions and electronic densities](https://mlsanity.com/paper/2106.02347/) (2021)
- [A Self-Attention Ansatz for Ab-initio Quantum Chemistry](https://mlsanity.com/paper/2211.13672/) (2022)
- [Ewald-based Long-Range Message Passing for Molecular Graphs](https://mlsanity.com/paper/2303.04791/) (2023)
- [So3krates: Equivariant attention for interactions on arbitrary length-scales in molecular systems](https://mlsanity.com/paper/2205.14276/) (2022)
- [Latent Ewald summation for machine learning of long-range interactions](https://mlsanity.com/paper/2408.15165/) (2024)
- [Physics-inspired Equivariant Descriptors of Non-bonded Interactions](https://mlsanity.com/paper/2308.13208/) (2023)
- [Completeness of Atomic Structure Representations](https://mlsanity.com/paper/2302.14770/) (2023)
- [BIGDML: Towards Exact Machine Learning Force Fields for Materials](https://mlsanity.com/paper/2106.04229/) (2021)

## Built on by (most cited first, 12 of 12)

- [Erwin: A Tree-based Hierarchical Transformer for Large-scale Physical Systems](https://mlsanity.com/paper/2502.17019/) (2025)
- [E2Former: An Efficient and Equivariant Transformer with Linear-Scaling Tensor Products](https://mlsanity.com/paper/2501.19216/) (2025)
- [Transferable Generative Models Bridge Femtosecond to Nanosecond Time-Step Molecular Dynamics](https://mlsanity.com/paper/2510.07589/) (2025)
- [Platonic Transformers: A Solid Choice For Equivariance](https://mlsanity.com/paper/2510.03511/) (2025)
- [A recipe for scalable attention-based MLIPs: unlocking long-range accuracy with all-to-all node attention](https://mlsanity.com/paper/2603.06567/) (2026)
- [chemtrain-deploy: A parallel and scalable framework for machine learning potentials in million-atom MD simulations](https://mlsanity.com/paper/2506.04055/) (2025)
- [Sampling 3D Molecular Conformers with Diffusion Transformers](https://mlsanity.com/paper/2506.15378/) (2025)
- [Learning Non-Local Molecular Interactions via Equivariant Local Representations and Charge Equilibration](https://mlsanity.com/paper/2501.19179/) (2025)
- [HollowFlow: Efficient Sample Likelihood Evaluation using Hollow Message Passing](https://mlsanity.com/paper/2510.21542/) (2025)
- [Generalization of Long-Range Machine Learning Potentials in Complex Chemical Spaces](https://mlsanity.com/paper/2512.10989/) (2025)
- [Reciprocal Space Attention for Learning Long-Range Interactions](https://mlsanity.com/paper/2510.13055/) (2025)
- [Physics-Informed Long-Range Coulomb Correction for Machine-learning Hamiltonians](https://mlsanity.com/paper/2603.20007/) (2026)

## Abstract (arXiv)

Long-range correlations are essential across numerous machine learning tasks, especially for data embedded in Euclidean space, where the relative positions and orientations of distant components are often critical for accurate predictions. Self-attention offers a compelling mechanism for capturing these global effects, but its quadratic complexity presents a significant practical limitation. This problem is particularly pronounced in computational chemistry, where the stringent efficiency requirements of machine learning force fields (MLFFs) often preclude accurately modeling long-range interactions. To address this, we introduce Euclidean fast attention (EFA), a linear-scaling attention-like mechanism designed for Euclidean data, which can be easily incorporated into existing model architectures. A core component of EFA are novel Euclidean rotary positional encodings (ERoPE), which enable efficient encoding of spatial information while respecting essential physical symmetries. We empirically demonstrate that EFA effectively captures diverse long-range effects, enabling EFA-equipped MLFFs to describe challenging chemical interactions for which conventional MLFFs yield incorrect results.