# Improving Equivariant Model Training via Constraint Relaxation

arXiv: https://arxiv.org/abs/2408.13242 · HTML: https://mlsanity.com/paper/2408.13242/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 60 references that are in the map.
- First posted on arXiv 23 Aug 2024.

## Builds on (most cited first, 30 of 41)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [General E(2)-Equivariant Steerable CNNs](https://mlsanity.com/paper/1911.08251/) (2019)
- [Invariant and Equivariant Graph Networks](https://mlsanity.com/paper/1812.09902/) (2018)
- [Vector Neurons: A General Framework for SO(3)-Equivariant Networks](https://mlsanity.com/paper/2104.12229/) (2021)
- [On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups](https://mlsanity.com/paper/1802.03690/) (2018)
- [Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs](https://mlsanity.com/paper/2206.11990/) (2022)
- [Learning SO(3) Equivariant Representations with Spherical CNNs](https://mlsanity.com/paper/1711.06721/) (2017)
- [A General Theory of Equivariant CNNs on Homogeneous Spaces](https://mlsanity.com/paper/1811.02017/) (2018)
- [Geometric and Physical Quantities Improve E(3) Equivariant Message Passing](https://mlsanity.com/paper/2110.02905/) (2021)
- [Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network](https://mlsanity.com/paper/1806.09231/) (2018)
- [Equivariance Through Parameter-Sharing](https://mlsanity.com/paper/1702.08389/) (2017)
- [Scalars are universal: Equivariant machine learning, structured like classical physics](https://mlsanity.com/paper/2106.06610/) (2021)
- [Lorentz Group Equivariant Neural Network for Particle Physics](https://mlsanity.com/paper/2006.04780/) (2020)
- [Reducing SO(3) Convolutions to SO(2) for Efficient Equivariant GNNs](https://mlsanity.com/paper/2302.03655/) (2023)
- [B-Spline CNNs on Lie Groups](https://mlsanity.com/paper/1909.12057/) (2019)
- [Approximately Equivariant Networks for Imperfectly Symmetric Dynamics](https://mlsanity.com/paper/2201.11969/) (2022)
- [Provably Strict Generalisation Benefit for Equivariant Models](https://mlsanity.com/paper/2102.10333/) (2021)
- [Sampling using SU(N) gauge equivariant flows](https://mlsanity.com/paper/2008.05456/) (2020)
- [Residual Pathway Priors for Soft Equivariance Constraints](https://mlsanity.com/paper/2112.01388/) (2021)
- [Equivariant Q Learning in Spatial Action Spaces](https://mlsanity.com/paper/2110.15443/) (2021)
- [The Lie Derivative for Measuring Learned Equivariance](https://mlsanity.com/paper/2210.02984/) (2022)
- [Sample Efficient Grasp Learning Using Equivariant Models](https://mlsanity.com/paper/2202.09468/) (2022)
- [Approximation-Generalization Trade-offs under (Approximate) Group Equivariance](https://mlsanity.com/paper/2305.17592/) (2023)
- [Learning Partial Equivariances from Data](https://mlsanity.com/paper/2110.10211/) (2021)
- [Relaxing Equivariance Constraints with Non-stationary Continuous Filters](https://mlsanity.com/paper/2204.07178/) (2022)
- [Equivariant Adaptation of Large Pretrained Models](https://mlsanity.com/paper/2310.01647/) (2023)
- [Enabling Efficient Equivariant Operations in the Fourier Basis via Gaunt Tensor Products](https://mlsanity.com/paper/2401.10216/) (2024)

## Built on by (most cited first, 23 of 23)

- [Relaxed Equivariance via Multitask Learning](https://mlsanity.com/paper/2410.17878/) (2024)
- [Probing Equivariance and Symmetry Breaking in Convolutional Networks](https://mlsanity.com/paper/2501.01999/) (2025)
- [Denoising Hamiltonian Network for Physical Reasoning](https://mlsanity.com/paper/2503.07596/) (2025)
- [Learning (Approximately) Equivariant Networks via Constrained Optimization](https://mlsanity.com/paper/2505.13631/) (2025)
- [Ensembles provably learn equivariance through data augmentation](https://mlsanity.com/paper/2410.01452/) (2024)
- [Flopping for FLOPs: Leveraging equivariance for computational efficiency](https://mlsanity.com/paper/2502.05169/) (2025)
- [A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Models](https://mlsanity.com/paper/2506.02269/) (2025)
- [Achieving Approximate Symmetry Is Exponentially Easier than Exact Symmetry](https://mlsanity.com/paper/2512.11855/) (2025)
- [Adaptive Canonicalization with Application to Invariant Anisotropic Geometric Networks](https://mlsanity.com/paper/2509.24886/) (2025)
- [Generalized Spherical Neural Operators: Green's Function Formulation](https://mlsanity.com/paper/2512.10723/) (2025)
- [Approximate Equivariance via Projection-based Regularisation](https://mlsanity.com/paper/2601.05028/) (2026)
- [How the Optimizer Shapes Learned Solutions in Equivariant Neural Networks](https://mlsanity.com/paper/2605.27662/) (2026)
- [Symmetries-enhanced Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2501.01136/) (2025)
- [Training Dynamics of Learning 3D-Rotational Equivariance](https://mlsanity.com/paper/2512.02303/) (2025)
- [Quasi-Equivariant Metanetworks](https://mlsanity.com/paper/2604.23720/) (2026)
- [SymDiff: Equivariant Diffusion via Stochastic Symmetrisation](https://mlsanity.com/paper/2410.06262/) (2024)
- [Drawback of Enforcing Equivariance and its Compensation via the Lens of Expressive Power](https://mlsanity.com/paper/2512.09673/) (2025)
- [Neural Operators for Biomedical Spherical Heterogeneity](https://mlsanity.com/paper/2601.03561/) (2026)
- [Recurrent Equivariant Constraint Modulation: Learning Per-Layer Symmetry Relaxation from Data](https://mlsanity.com/paper/2602.02853/) (2026)
- [Multi-Task Reinforcement Learning of Drone Aerobatics by Exploiting Geometric Symmetries](https://mlsanity.com/paper/2602.10997/) (2026)
- [Scale-invariant Gaussian derivative residual networks](https://mlsanity.com/paper/2603.02843/) (2026)
- [Rotational Equivariance in Machine Learning: A Comprehensive Tutorial](https://mlsanity.com/paper/2608.31045/) (2026)
- [Equivariance Breaks the Learning Rate](https://mlsanity.com/paper/2609.08381/) (2026)

## Abstract (arXiv)

Equivariant neural networks have been widely used in a variety of applications due to their ability to generalize well in tasks where the underlying data symmetries are known. Despite their successes, such networks can be difficult to optimize and require careful hyperparameter tuning to train successfully. In this work, we propose a novel framework for improving the optimization of such models by relaxing the hard equivariance constraint during training: We relax the equivariance constraint of the network's intermediate layers by introducing an additional non-equivariant term that we progressively constrain until we arrive at an equivariant solution. By controlling the magnitude of the activation of the additional relaxation term, we allow the model to optimize over a larger hypothesis space containing approximate equivariant networks and converge back to an equivariant solution at the end of training. We provide experimental results on different state-of-the-art network architectures, demonstrating how this training framework can result in equivariant models with improved generalization performance. Our code is available at https://github.com/StefanosPert/Equivariant_Optimization_CR