# Learning Probabilistic Symmetrization for Architecture Agnostic Equivariance

arXiv: https://arxiv.org/abs/2306.02866 · HTML: https://mlsanity.com/paper/2306.02866/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 95 of its 109 references that are in the map.
- First posted on arXiv 5 Jun 2023.

## Builds on (most cited first, 30 of 95)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering](https://mlsanity.com/paper/1606.09375/) (2016)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [E(n) Equivariant Graph Neural Networks](https://mlsanity.com/paper/2102.09844/) (2021)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networks](https://mlsanity.com/paper/1810.02244/) (2018)
- [Simple and Deep Graph Convolutional Networks](https://mlsanity.com/paper/2007.02133/) (2020)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [Recipe for a General, Powerful, Scalable Graph Transformer](https://mlsanity.com/paper/2205.12454/) (2022)
- [Benchmarking Graph Neural Networks](https://mlsanity.com/paper/2003.00982/) (2020)

## Built on by (most cited first, 26 of 26)

- [Equivariant Frames and the Impossibility of Continuous Canonicalization](https://mlsanity.com/paper/2402.16077/) (2024)
- [Equivariance via Minimal Frame Averaging for More Symmetries and Efficiency](https://mlsanity.com/paper/2406.07598/) (2024)
- [Improving Equivariant Networks with Probabilistic Symmetry Breaking](https://mlsanity.com/paper/2503.21985/) (2025)
- [Probing the effects of broken symmetries in machine learning](https://mlsanity.com/paper/2406.17747/) (2024)
- [Revisiting Random Walks for Learning on Graphs](https://mlsanity.com/paper/2407.01214/) (2024)
- [Relaxed Equivariance via Multitask Learning](https://mlsanity.com/paper/2410.17878/) (2024)
- [On the Completeness of Invariant Geometric Deep Learning Models](https://mlsanity.com/paper/2402.04836/) (2024)
- [Weisfeiler Leman for Euclidean Equivariant Machine Learning](https://mlsanity.com/paper/2402.02484/) (2024)
- [A Generative Model of Symmetry Transformations](https://mlsanity.com/paper/2403.01946/) (2024)
- [Approximately Equivariant Neural Processes](https://mlsanity.com/paper/2406.13488/) (2024)
- [Improved Canonicalization for Model Agnostic Equivariance](https://mlsanity.com/paper/2405.14089/) (2024)
- [Koopman-Equivariant Gaussian Processes](https://mlsanity.com/paper/2502.06645/) (2025)
- [Learning Symmetrization for Equivariance with Orbit Distance Minimization](https://mlsanity.com/paper/2311.07143/) (2023)
- [Symmetry-Aware Generative Modeling through Learned Canonicalization](https://mlsanity.com/paper/2501.07773/) (2025)
- [Scaling Laws and Symmetry, Evidence from Neural Force Fields](https://mlsanity.com/paper/2510.09768/) (2025)
- [CP2: Leveraging Geometry for Conformal Prediction via Canonicalization](https://mlsanity.com/paper/2506.16189/) (2025)
- [Inverting Data Transformations via Diffusion Sampling](https://mlsanity.com/paper/2602.08267/) (2026)
- [Efficient Model-Agnostic Multi-Group Equivariant Networks](https://mlsanity.com/paper/2310.09675/) (2023)
- [G-RepsNet: A Fast and General Construction of Equivariant Networks for Arbitrary Matrix Groups](https://mlsanity.com/paper/2402.15413/) (2024)
- [On Equivariant Model Selection through the Lens of Uncertainty](https://mlsanity.com/paper/2506.18629/) (2025)
- [SAVLA: Symmetry-Aware Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2609.16641/) (2026)
- [Adaptive Sampling for Continuous Group Equivariant Neural Networks](https://mlsanity.com/paper/2409.08741/) (2024)
- [Symmetry From Scratch: Group Equivariance as a Supervised Learning Task](https://mlsanity.com/paper/2410.03989/) (2024)
- [SymDiff: Equivariant Diffusion via Stochastic Symmetrisation](https://mlsanity.com/paper/2410.06262/) (2024)
- [Group-Equivariant Poincar'e Convolutional Networks](https://mlsanity.com/paper/2607.00556/) (2026)
- [PATTERN Can Be Solved from Memory: Generator-Aware Baselines for Synthetic Node Classification](https://mlsanity.com/paper/2610.15020/) (2026)

## Abstract (arXiv)

We present a novel framework to overcome the limitations of equivariant architectures in learning functions with group symmetries. In contrary to equivariant architectures, we use an arbitrary base model such as an MLP or a transformer and symmetrize it to be equivariant to the given group by employing a small equivariant network that parameterizes the probabilistic distribution underlying the symmetrization. The distribution is end-to-end trained with the base model which can maximize performance while reducing sample complexity of symmetrization. We show that this approach ensures not only equivariance to given group but also universal approximation capability in expectation. We implement our method on various base models, including patch-based transformers that can be initialized from pretrained vision transformers, and test them for a wide range of symmetry groups including permutation and Euclidean groups and their combinations. Empirical tests show competitive results against tailored equivariant architectures, suggesting the potential for learning equivariant functions for diverse groups using a non-equivariant universal base architecture. We further show evidence of enhanced learning in symmetric modalities, like graphs, when pretrained from non-symmetric modalities, like vision. Code is available at https://github.com/jw9730/lps.