# Equivariant Transformer is all you need

arXiv: https://arxiv.org/abs/2310.13222 · HTML: https://mlsanity.com/paper/2310.13222/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 17 references that are in the map.
- First posted on arXiv 20 Oct 2023.

## Builds on (most cited first, 5 of 5)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials](https://mlsanity.com/paper/2101.03164/) (2021)
- [TorchMD-NET: Equivariant Transformers for Neural Network based Molecular Potentials](https://mlsanity.com/paper/2202.02541/) (2022)
- [Advances in machine-learning-based sampling motivated by lattice quantum chromodynamics](https://mlsanity.com/paper/2309.01156/) (2023)

## Built on by (most cited first, 3 of 3)

- [Masked Particle Modeling on Sets: Towards Self-Supervised High Energy Physics Foundation Models](https://mlsanity.com/paper/2401.13537/) (2024)
- [Physics-Driven Learning for Inverse Problems in Quantum Chromodynamics](https://mlsanity.com/paper/2501.05580/) (2025)
- [Generating particle physics Lagrangians with transformers](https://mlsanity.com/paper/2501.09729/) (2025)

## Abstract (arXiv)

Machine learning, deep learning, has been accelerating computational physics, which has been used to simulate systems on a lattice. Equivariance is essential to simulate a physical system because it imposes a strong induction bias for the probability distribution described by a machine learning model. This reduces the risk of erroneous extrapolation that deviates from data symmetries and physical laws. However, imposing symmetry on the model sometimes occur a poor acceptance rate in self-learning Monte-Carlo (SLMC). On the other hand, Attention used in Transformers like GPT realizes a large model capacity. We introduce symmetry equivariant attention to SLMC. To evaluate our architecture, we apply it to our proposed new architecture on a spin-fermion model on a two-dimensional lattice. We find that it overcomes poor acceptance rates for linear models and observe the scaling law of the acceptance rate as in the large language models with Transformers.