# Exploring Weight Symmetry in Deep Neural Networks

arXiv: https://arxiv.org/abs/1812.11027 · HTML: https://mlsanity.com/paper/1812.11027/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Dec 2018.

## Built on by (most cited first, 12 of 12)

- [Understanding convolution on graphs via energies](https://mlsanity.com/paper/2206.10991/) (2022)
- [Transformers from an Optimization Perspective](https://mlsanity.com/paper/2205.13891/) (2022)
- [Generalized energy and gradient flow via graph framelets](https://mlsanity.com/paper/2210.04124/) (2022)
- [Tilt your Head: Activating the Hidden Spatial-Invariance of Classifiers](https://mlsanity.com/paper/2405.03730/) (2024)
- [Hierarchical Associative Memory, Parallelized MLP-Mixer, and Symmetry Breaking](https://mlsanity.com/paper/2406.12220/) (2024)
- [Learning Continuous Rotation Canonicalization with Radial Beam Sampling](https://mlsanity.com/paper/2206.10690/) (2022)
- [Discrete Cosine Transform Based Decorrelated Attention for Vision Transformers](https://mlsanity.com/paper/2405.13901/) (2024)
- [Switchable Lightweight Anti-symmetric Processing (SLAP) with CNN Outspeeds Data Augmentation by Smaller Sample -- Application in Gomoku Reinforcement Learning](https://mlsanity.com/paper/2301.04746/) (2023)
- [Benefits of mirror weight symmetry for 3D mesh segmentation in biomedical applications](https://mlsanity.com/paper/2309.17076/) (2023)
- [Towards Symmetric Low-Rank Adapters](https://mlsanity.com/paper/2504.03719/) (2025)
- [Binding threshold units with artificial oscillatory neurons](https://mlsanity.com/paper/2505.03648/) (2025)
- [Symmetry and Generalisation in Neural Approximations of Renormalisation Transformations](https://mlsanity.com/paper/2510.16591/) (2025)

## Abstract (arXiv)

We propose to impose symmetry in neural network parameters to improve parameter usage and make use of dedicated convolution and matrix multiplication routines. Due to significant reduction in the number of parameters as a result of the symmetry constraints, one would expect a dramatic drop in accuracy. Surprisingly, we show that this is not the case, and, depending on network size, symmetry can have little or no negative effect on network accuracy, especially in deep overparameterized networks. We propose several ways to impose local symmetry in recurrent and convolutional neural networks, and show that our symmetry parameterizations satisfy universal approximation property for single hidden layer networks. We extensively evaluate these parameterizations on CIFAR, ImageNet and language modeling datasets, showing significant benefits from the use of symmetry. For instance, our ResNet-101 with channel-wise symmetry has almost 25% less parameters and only 0.2% accuracy loss on ImageNet. Code for our experiments is available at https://github.com/hushell/deep-symmetry