# Weight-Sharing Regularization

arXiv: https://arxiv.org/abs/2311.03096 · HTML: https://mlsanity.com/paper/2311.03096/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 39 references that are in the map.
- First posted on arXiv 6 Nov 2023.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Equivariance Through Parameter-Sharing](https://mlsanity.com/paper/1702.08389/) (2017)
- [Soft Weight-Sharing for Neural Network Compression](https://mlsanity.com/paper/1702.04008/) (2017)
- [Meta-Learning Symmetries by Reparameterization](https://mlsanity.com/paper/2007.02933/) (2020)
- [Towards Learning Convolutions from Scratch](https://mlsanity.com/paper/2007.13657/) (2020)
- [Equivariance Discovery by Learned Parameter-Sharing](https://mlsanity.com/paper/2204.03640/) (2022)
- [spred: Solving L1 Penalty with SGD](https://mlsanity.com/paper/2210.01212/) (2022)
- [Efficient Implementations of the Generalized Lasso Dual Path Algorithm](https://mlsanity.com/paper/1405.3222/) (2014)

## Built on by (most cited first, 2 of 2)

- [MCNC: Manifold-Constrained Reparameterization for Neural Compression](https://mlsanity.com/paper/2406.19301/) (2024)
- [Group-Equivariant Poincar'e Convolutional Networks](https://mlsanity.com/paper/2607.00556/) (2026)

## Abstract (arXiv)

Weight-sharing is ubiquitous in deep learning. Motivated by this, we propose a "weight-sharing regularization" penalty on the weights w ∈ ℝd of a neural network, defined as R(w) = frac1d - 1sumi > jd |wi - wj|. We study the proximal mapping of R and provide an intuitive interpretation of it in terms of a physical system of interacting particles. We also parallelize existing algorithms for operatornameproxR (to run on GPU) and find that one of them is fast in practice but slow (O(d)) for worst-case inputs. Using the physical interpretation, we design a novel parallel algorithm which runs in O(log3 d) when sufficient processors are available, thus guaranteeing fast training. Our experiments reveal that weight-sharing regularization enables fully connected networks to learn convolution-like filters even when pixels have been shuffled while convolutional neural networks fail in this setting. Our code is available on github.