mlsanityData to 14 Oct 2026

Does equivariance matter at scale?

arXiv 2410.23179 ↗First posted 30 Oct 2024cs.LGcited by 52

Not on the map. Why?

Abstract (arXiv)

Given large datasets and sufficient compute, is it beneficial to design neural architectures for the structure and symmetries of each problem? Or is it more efficient to learn them from data? We study empirically how equivariant and non-equivariant networks scale with compute and training samples. Focusing on a benchmark problem of rigid-body interactions and on general-purpose transformer architectures, we perform a series of experiments, varying the model size, training steps, and dataset size. We find evidence for three conclusions. First, equivariance improves data efficiency, but training non-equivariant models with data augmentation can close this gap given sufficient epochs. Second, scaling with compute follows a power law, with equivariant models outperforming non-equivariant ones at each tested compute budget. Finally, the optimal allocation of a compute budget onto model size and training duration differs between equivariant and non-equivariant models.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Builds on 62 of its 79 references that are in the map. First posted on arXiv 30 Oct 2024.

Builds on

62 of 79 references.

References · 62

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  7. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  8. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,167
  9. 2016Group Equivariant Convolutional Networkscited by 1,165
  10. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  11. 2021E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentialscited by 708
  12. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
38 more
  1. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  2. 2022MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fieldscited by 474
  3. 2018Rotation Equivariant CNNs for Digital Pathologycited by 422
  4. 2023Scaling Data-Constrained Language Modelscited by 403
  5. 2022Kubric: A scalable dataset generatorcited by 381
  6. 2021Explaining Neural Scaling Lawscited by 363
  7. 2022Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphscited by 264
  8. 2022Learning Local Equivariant Representations for Large-Scale Atomistic Dynamicscited by 256
  9. 2021Geometric and Physical Quantities Improve E(3) Equivariant Message Passingcited by 228
  10. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  11. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 210
  12. 2019Ab-Initio Solution of the Many-Electron Schr"odinger Equation with Deep Neural Networkscited by 200
  13. 2019Deep neural network solution of the electronic Schr"odinger equationcited by 127
  14. 2022Equivariant 3D-Conditional Diffusion Models for Molecular Linker Designcited by 122
  15. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  16. 2023MatterGen: a generative model for inorganic materials designcited by 98
  17. 2018Roto-Translation Covariant Convolutional Networks for Medical Image Analysiscited by 92
  18. 2021Provably Strict Generalisation Benefit for Equivariant Modelscited by 91
  19. 2023Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Designcited by 89
  20. 2020Sampling using SU(N) gauge equivariant flowscited by 84
  21. 2022SO(2)-Equivariant Reinforcement Learningcited by 84
  22. 2020On the Benefits of Invariance in Neural Networkscited by 79
  23. 2021Scaling Scaling Laws with Board Gamescited by 63
  24. 2021Equivariant Q Learning in Spatial Action Spacescited by 62
  25. 2023Clifford Group Equivariant Neural Networkscited by 55
  26. 2023Geometric Algebra Transformercited by 48
  27. 2023Approximation-Generalization Trade-offs under (Approximate) Group Equivariancecited by 46
  28. 2022Learning rigid dynamics with face interaction graph networkscited by 43
  29. 2016Generalization Error of Invariant Classifierscited by 42
  30. 2022On-Robot Learning With Equivariant Modelscited by 42
  31. 2020ContactNets: Learning Discontinuous Contact Dynamics with Smooth, Implicit Representationscited by 38
  32. 2023Geometric Clifford Algebra Networkscited by 38
  33. 2024Lorentz-Equivariant Geometric Algebra Transformers for High-Energy Physicscited by 38
  34. 2022Learning Lattice Quantum Field Theories with Equivariant Continuous Flowscited by 37
  35. 20183D G-CNNs for Pulmonary Nodule Detectioncited by 36
  36. 2023EDGI: Equivariant Diffusion for Planning with Embodied Agentscited by 36
  37. 2019Improved Generalization Bounds of Group Invariant / Equivariant Deep Networks via Quotient Feature Spacescited by 34
  38. 2022Mesh Neural Networks for SE(3)-Equivariant Hemodynamics Estimation on the Artery Wallcited by 29

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 24; 2026: 27.

52 papers.

Sort:

By area:

  1. 2025UMA: A Family of Universal Models for Atomscited by 105
  2. 2025EQ-VAE: Equivariance Regularized Latent Space for Improved Generative Image Modelingcited by 89
  3. 2025Learning Smooth and Expressive Interatomic Potentials for Physical Property Predictioncited by 84
  4. 2025Position: Beyond Euclidean -- Foundation Models Should Embrace Non-Euclidean Geometriescited by 18
  5. 2025Probing Equivariance and Symmetry Breaking in Convolutional Networkscited by 12
  6. 2025Lorentz Local Canonicalization: How to Make Any Network Lorentz-Equivariantcited by 12
  7. 2024On the Ability of Deep Networks to Learn Symmetries from Data: A Neural Kernel Theorycited by 10
  8. 2025How simple can you go? An off-the-shelf transformer approach to molecular dynamicscited by 10
  9. 2025ELECTRA: A Cartesian Network for 3D Charge Density Prediction with Floating Orbitalscited by 10
  10. 2025Flopping for FLOPs: Leveraging equivariance for computational efficiencycited by 8
  11. 2025GradMetaNet: An Equivariant Architecture for Learning on Gradientscited by 7
  12. 2026Identifiable Equivariant Networks are Layerwise Equivariantcited by 7
38 more
  1. 2025When is a System Discoverable from Data? Discovery Requires Chaoscited by 6
  2. 2026Scaling laws for amplitude surrogatescited by 6
  3. 2025Rao-Blackwell Gradient Estimators for Equivariant Denoising Diffusioncited by 5
  4. 2025LeMat-Traj: A Scalable and Unified Dataset of Materials Trajectories for Atomistic Modelingcited by 4
  5. 2025Scaling Laws and Symmetry, Evidence from Neural Force Fieldscited by 4
  6. 2025Quick ViTs: Speeding up Vision Transformers through Equivariancecited by 3
  7. 2025A Circular Argument : Does RoPE need to be Equivariant for Vision?cited by 3
  8. 2025AdS-GNN -- a Conformally Equivariant Graph Neural Networkcited by 2
  9. 2025CP2: Leveraging Geometry for Conformal Prediction via Canonicalizationcited by 2
  10. 2026Approximate Equivariance via Projection-based Regularisationcited by 2
  11. 2026A Graph Meta-Network for Learning on Kolmogorov-Arnold Networkscited by 2
  12. 2026Statistical Properties of Training & Generalizationcited by 2
  13. 2025Equivariance by Local Canonicalization: A Matter of Representationcited by 1
  14. 2025On Uncertainty Calibration for Equivariant Functionscited by 1
  15. 2025Training Dynamics of Learning 3D-Rotational Equivariancecited by 1
  16. 2026Quantitative Approximation Rates for Group Equivariant Learningcited by 1
  17. 2026Conformal Orbit-Valid Trust Horizons for Equivariant World Modelscited by 1
  18. 2025Auto-encoding Molecules: Graph-Matching Capabilities Mattercited by 0
  19. 2025Conditional Clifford-Steerable CNNs for PDE Modelingcited by 0
  20. 2025Soft Geometric Inductive Bias for Object Centric Dynamicscited by 0
  21. 2026Balancing Symmetry and Efficiency in Graph Flow Matchingcited by 0
  22. 2026Position: Stop Preaching and Start Practising Data Frugality for Responsible Development of AIcited by 0
  23. 2026Coupled Cluster con M=oLe: Molecular Orbital Learning for Neural Wavefunctionscited by 0
  24. 2026Accurate Shift Invariant Convolutional Neural Networks Using Gaussian-Hermite Momentscited by 0
  25. 2026Learning Coordinate-based Convolutional Kernels for Continuous SE(3) Equivariant and Efficient Point Cloud Analysiscited by 0
  26. 2026When Does Equivariance Help? Canonical Alignment in Neural Fluid Surrogatescited by 0
  27. 2026Towards Distillation Guarantees under Algorithmic Alignment for Combinatorial Optimizationcited by 0
  28. 2026Equivariance and Augmentation for Bayesian Neural Networkscited by 0
  29. 2026MALOQ: Massively Accelerated Learning of Operators for Quantum Transportcited by 0
  30. 2026Boosting Data Augmentation with Stochastic Weight Averagingcited by 0
  31. 2026Rotational Equivariance in Machine Learning: A Comprehensive Tutorialcited by 0
  32. 2026Equivariance Breaks the Learning Ratecited by 0
  33. 2026Transformation Laws in Neural Representations: Structure, Realisability, and Constructioncited by 0
  34. 2026EquiSELD: Efficient training of equivariant sound event localization and detection networkscited by 0
  35. 2026EMERGE: Resolution-Agnostic Point Cloud Generation with Equivariant Graph-Based Diffusioncited by 0
  36. 2026Permutation-Equivariant Flow Matching for Alignment-Free Neural Weight Generationcited by 0
  37. 2026A library for differentiable signal processing and machine learning on the spherecited by 0
  38. 2026EDISCO: Equivariant DIScrete Diffusion for Euclidean Combinatorial Optimizationcited by 0

50 most cited of 52 All 52 (JSON)