# Bespoke vs. Pret-`a-Porter Lottery Tickets: Exploiting Mask Similarity for Trainable Sub-Network Finding

arXiv: https://arxiv.org/abs/2007.04091 · HTML: https://mlsanity.com/paper/2007.04091/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jul 2020.

## Built on by (most cited first, 5 of 5)

- [DiSparse: Disentangled Sparsification for Multitask Model Compression](https://mlsanity.com/paper/2206.04662/) (2022)
- [Bridging Lottery Ticket and Grokking: Understanding Grokking from Inner Structure of Networks](https://mlsanity.com/paper/2310.19470/) (2023)
- [Towards Compute-Optimal Transfer Learning](https://mlsanity.com/paper/2304.13164/) (2023)
- [Studying the Consistency and Composability of Lottery Ticket Pruning Masks](https://mlsanity.com/paper/2104.14753/) (2021)
- [Distilled Pruning: Using Synthetic Data to Win the Lottery](https://mlsanity.com/paper/2307.03364/) (2023)

## Abstract (arXiv)

The observation of sparse trainable sub-networks within over-parametrized networks - also known as Lottery Tickets (LTs) - has prompted inquiries around their trainability, scaling, uniqueness, and generalization properties. Across 28 combinations of image classification tasks and architectures, we discover differences in the connectivity structure of LTs found through different iterative pruning techniques, thus disproving their uniqueness and connecting emergent mask structure to the choice of pruning. In addition, we propose a consensus-based method for generating refined lottery tickets. This lottery ticket denoising procedure, based on the principle that parameters that always go unpruned across different tasks more reliably identify important sub-networks, is capable of selecting a meaningful portion of the architecture in an embarrassingly parallel way, while quickly discarding extra parameters without the need for further pruning iterations. We successfully train these sub-networks to performance comparable to that of ordinary lottery tickets.