# Does equivariance matter at scale?

arXiv: https://arxiv.org/abs/2410.23179 · HTML: https://mlsanity.com/paper/2410.23179/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Builds on 62 of its 79 references that are in the map.
- First posted on arXiv 30 Oct 2024.

## Builds on (most cited first, 30 of 62)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials](https://mlsanity.com/paper/2101.03164/) (2021)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fields](https://mlsanity.com/paper/2206.07697/) (2022)
- [Rotation Equivariant CNNs for Digital Pathology](https://mlsanity.com/paper/1806.03962/) (2018)
- [Scaling Data-Constrained Language Models](https://mlsanity.com/paper/2305.16264/) (2023)
- [Kubric: A scalable dataset generator](https://mlsanity.com/paper/2203.03570/) (2022)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs](https://mlsanity.com/paper/2206.11990/) (2022)
- [Learning Local Equivariant Representations for Large-Scale Atomistic Dynamics](https://mlsanity.com/paper/2204.05249/) (2022)
- [Geometric and Physical Quantities Improve E(3) Equivariant Message Passing](https://mlsanity.com/paper/2110.02905/) (2021)
- [Adaptive Input Representations for Neural Language Modeling](https://mlsanity.com/paper/1809.10853/) (2018)
- [A Constructive Prediction of the Generalization Error Across Scales](https://mlsanity.com/paper/1909.12673/) (2019)
- [Ab-Initio Solution of the Many-Electron Schr"odinger Equation with Deep Neural Networks](https://mlsanity.com/paper/1909.02487/) (2019)
- [Deep neural network solution of the electronic Schr"odinger equation](https://mlsanity.com/paper/1909.08423/) (2019)
- [Equivariant 3D-Conditional Diffusion Models for Molecular Linker Design](https://mlsanity.com/paper/2210.05274/) (2022)
- [Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?](https://mlsanity.com/paper/2207.10551/) (2022)
- [MatterGen: a generative model for inorganic materials design](https://mlsanity.com/paper/2312.03687/) (2023)
- [Roto-Translation Covariant Convolutional Networks for Medical Image Analysis](https://mlsanity.com/paper/1804.03393/) (2018)
- [Provably Strict Generalisation Benefit for Equivariant Models](https://mlsanity.com/paper/2102.10333/) (2021)

## Built on by (most cited first, 30 of 52)

- [UMA: A Family of Universal Models for Atoms](https://mlsanity.com/paper/2506.23971/) (2025)
- [EQ-VAE: Equivariance Regularized Latent Space for Improved Generative Image Modeling](https://mlsanity.com/paper/2502.09509/) (2025)
- [Learning Smooth and Expressive Interatomic Potentials for Physical Property Prediction](https://mlsanity.com/paper/2502.12147/) (2025)
- [Position: Beyond Euclidean -- Foundation Models Should Embrace Non-Euclidean Geometries](https://mlsanity.com/paper/2504.08896/) (2025)
- [Probing Equivariance and Symmetry Breaking in Convolutional Networks](https://mlsanity.com/paper/2501.01999/) (2025)
- [Lorentz Local Canonicalization: How to Make Any Network Lorentz-Equivariant](https://mlsanity.com/paper/2505.20280/) (2025)
- [On the Ability of Deep Networks to Learn Symmetries from Data: A Neural Kernel Theory](https://mlsanity.com/paper/2412.11521/) (2024)
- [How simple can you go? An off-the-shelf transformer approach to molecular dynamics](https://mlsanity.com/paper/2503.01431/) (2025)
- [ELECTRA: A Cartesian Network for 3D Charge Density Prediction with Floating Orbitals](https://mlsanity.com/paper/2503.08305/) (2025)
- [Flopping for FLOPs: Leveraging equivariance for computational efficiency](https://mlsanity.com/paper/2502.05169/) (2025)
- [GradMetaNet: An Equivariant Architecture for Learning on Gradients](https://mlsanity.com/paper/2507.01649/) (2025)
- [Identifiable Equivariant Networks are Layerwise Equivariant](https://mlsanity.com/paper/2601.21645/) (2026)
- [When is a System Discoverable from Data? Discovery Requires Chaos](https://mlsanity.com/paper/2511.08860/) (2025)
- [Scaling laws for amplitude surrogates](https://mlsanity.com/paper/2601.13308/) (2026)
- [Rao-Blackwell Gradient Estimators for Equivariant Denoising Diffusion](https://mlsanity.com/paper/2502.09890/) (2025)
- [LeMat-Traj: A Scalable and Unified Dataset of Materials Trajectories for Atomistic Modeling](https://mlsanity.com/paper/2508.20875/) (2025)
- [Scaling Laws and Symmetry, Evidence from Neural Force Fields](https://mlsanity.com/paper/2510.09768/) (2025)
- [Quick ViTs: Speeding up Vision Transformers through Equivariance](https://mlsanity.com/paper/2505.15441/) (2025)
- [A Circular Argument : Does RoPE need to be Equivariant for Vision?](https://mlsanity.com/paper/2511.08368/) (2025)
- [AdS-GNN -- a Conformally Equivariant Graph Neural Network](https://mlsanity.com/paper/2505.12880/) (2025)
- [CP2: Leveraging Geometry for Conformal Prediction via Canonicalization](https://mlsanity.com/paper/2506.16189/) (2025)
- [Approximate Equivariance via Projection-based Regularisation](https://mlsanity.com/paper/2601.05028/) (2026)
- [A Graph Meta-Network for Learning on Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2602.16316/) (2026)
- [Statistical Properties of Training & Generalization](https://mlsanity.com/paper/2606.20299/) (2026)
- [Equivariance by Local Canonicalization: A Matter of Representation](https://mlsanity.com/paper/2509.26499/) (2025)
- [On Uncertainty Calibration for Equivariant Functions](https://mlsanity.com/paper/2510.21691/) (2025)
- [Training Dynamics of Learning 3D-Rotational Equivariance](https://mlsanity.com/paper/2512.02303/) (2025)
- [Quantitative Approximation Rates for Group Equivariant Learning](https://mlsanity.com/paper/2602.20370/) (2026)
- [Conformal Orbit-Valid Trust Horizons for Equivariant World Models](https://mlsanity.com/paper/2606.24946/) (2026)
- [Auto-encoding Molecules: Graph-Matching Capabilities Matter](https://mlsanity.com/paper/2503.00426/) (2025)

## Abstract (arXiv)

Given large datasets and sufficient compute, is it beneficial to design neural architectures for the structure and symmetries of each problem? Or is it more efficient to learn them from data? We study empirically how equivariant and non-equivariant networks scale with compute and training samples. Focusing on a benchmark problem of rigid-body interactions and on general-purpose transformer architectures, we perform a series of experiments, varying the model size, training steps, and dataset size. We find evidence for three conclusions. First, equivariance improves data efficiency, but training non-equivariant models with data augmentation can close this gap given sufficient epochs. Second, scaling with compute follows a power law, with equivariant models outperforming non-equivariant ones at each tested compute budget. Finally, the optimal allocation of a compute budget onto model size and training duration differs between equivariant and non-equivariant models.