# Signing the Supermask: Keep, Hide, Invert

arXiv: https://arxiv.org/abs/2201.13361 · HTML: https://mlsanity.com/paper/2201.13361/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 31 Jan 2022.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [Importance Estimation for Neural Network Pruning](https://mlsanity.com/paper/1906.10771/) (2019)
- [Rigging the Lottery: Making All Tickets Winners](https://mlsanity.com/paper/1911.11134/) (2019)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science](https://mlsanity.com/paper/1707.04780/) (2017)
- [What's Hidden in a Randomly Weighted Neural Network?](https://mlsanity.com/paper/1911.13299/) (2019)
- [Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask](https://mlsanity.com/paper/1905.01067/) (2019)
- [Ternary Weight Networks](https://mlsanity.com/paper/1605.04711/) (2016)
- [Sparse Training via Boosting Pruning Plasticity with Neuroregeneration](https://mlsanity.com/paper/2106.10404/) (2021)
- [Progressive Skeletonization: Trimming more fat from a network at initialization](https://mlsanity.com/paper/2006.09081/) (2020)
- [Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network](https://mlsanity.com/paper/2103.09377/) (2021)
- [Ternary Neural Networks for Resource-Efficient AI Applications](https://mlsanity.com/paper/1609.00222/) (2016)
- [Learning Discrete Weights Using the Local Reparameterization Trick](https://mlsanity.com/paper/1710.07739/) (2017)
- [Pruning Randomly Initialized Neural Networks with Iterative Randomization](https://mlsanity.com/paper/2106.09269/) (2021)
- [Balanced Binary Neural Networks with Gated Residual](https://mlsanity.com/paper/1909.12117/) (2019)
- [Deep Residual Networks with Exponential Linear Unit](https://mlsanity.com/paper/1604.04112/) (2016)

## Built on by (most cited first, 7 of 7)

- [Lifelong Reinforcement Learning with Modulating Masks](https://mlsanity.com/paper/2212.11110/) (2022)
- [Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey](https://mlsanity.com/paper/2205.08099/) (2022)
- [Sharing Lifelong Reinforcement Learning Knowledge via Modulating Masks](https://mlsanity.com/paper/2305.10997/) (2023)
- [Masked Random Noise for Communication Efficient Federated Learning](https://mlsanity.com/paper/2408.03220/) (2024)
- [Find A Winning Sign: Sign Is All We Need to Win the Lottery](https://mlsanity.com/paper/2504.05357/) (2025)
- [Policy Search, Retrieval, and Composition via Task Similarity in Collaborative Agentic Systems](https://mlsanity.com/paper/2506.05577/) (2025)
- [How to Find and Reuse Policies for Continuous Adaptation in Lifelong Reinforcement Learning](https://mlsanity.com/paper/2610.03119/) (2026)

## Abstract (arXiv)

The exponential growth in numbers of parameters of neural networks over the past years has been accompanied by an increase in performance across several fields. However, due to their sheer size, the networks not only became difficult to interpret but also problematic to train and use in real-world applications, since hardware requirements increased accordingly. Tackling both issues, we present a novel approach that either drops a neural network's initial weights or inverts their respective sign. Put simply, a network is trained by weight selection and inversion without changing their absolute values. Our contribution extends previous work on masking by additionally sign-inverting the initial weights and follows the findings of the Lottery Ticket Hypothesis. Through this extension and adaptations of initialization methods, we achieve a pruning rate of up to 99%, while still matching or exceeding the performance of various baseline and previous models. Our approach has two main advantages. First, and most notable, signed Supermask models drastically simplify a model's structure, while still performing well on given tasks. Second, by reducing the neural network to its very foundation, we gain insights into which weights matter for performance. The code is available on GitHub.