# Neural Parameter Allocation Search

arXiv: https://arxiv.org/abs/2006.10598 · HTML: https://mlsanity.com/paper/2006.10598/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2020.

## Built on by (most cited first, 15 of 15)

- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [CHAMMI: A benchmark for channel-adaptive models in microscopy imaging](https://mlsanity.com/paper/2310.19224/) (2023)
- [Neural Network Architecture Beyond Width and Depth](https://mlsanity.com/paper/2205.09459/) (2022)
- [InstructDET: Diversifying Referring Object Detection with Generalized Instructions](https://mlsanity.com/paper/2310.05136/) (2023)
- [Enhancing Feature Diversity Boosts Channel-Adaptive Vision Transformers](https://mlsanity.com/paper/2405.16419/) (2024)
- [MixtureGrowth: Growing Neural Networks by Recombining Learned Parameters](https://mlsanity.com/paper/2311.04251/) (2023)
- [On Enhancing Expressive Power via Compositions of Single Fixed-Size ReLU Network](https://mlsanity.com/paper/2301.12353/) (2023)
- [ChA-MAEViT: Unifying Channel-Aware Masked Autoencoders and Multi-Channel Vision Transformers for Improved Cross-Channel Learning](https://mlsanity.com/paper/2503.19331/) (2025)
- [Homological Convolutional Neural Networks](https://mlsanity.com/paper/2308.13816/) (2023)
- [MCNC: Manifold-Constrained Reparameterization for Neural Compression](https://mlsanity.com/paper/2406.19301/) (2024)
- [RECAST: Reparameterized, Compact weight Adaptation for Sequential Tasks](https://mlsanity.com/paper/2411.16870/) (2024)
- [STen: Productive and Efficient Sparsity in PyTorch](https://mlsanity.com/paper/2304.07613/) (2023)
- [Learning to Compose SuperWeights for Neural Parameter Allocation Search](https://mlsanity.com/paper/2312.01274/) (2023)
- [On Information Geometry and Iterative Optimization in Model Compression: Operator Factorization](https://mlsanity.com/paper/2507.09428/) (2025)
- [Decompose, Mix, Adapt: A Unified Framework for Parameter-Efficient Neural Network Recombination and Compression](https://mlsanity.com/paper/2603.27383/) (2026)

## Abstract (arXiv)

Training neural networks requires increasing amounts of memory. Parameter sharing can reduce memory and communication costs, but existing methods assume networks have many identical layers and utilize hand-crafted sharing strategies that fail to generalize. We introduce Neural Parameter Allocation Search (NPAS), a novel task where the goal is to train a neural network given an arbitrary, fixed parameter budget. NPAS covers both low-budget regimes, which produce compact networks, as well as a novel high-budget regime, where additional capacity can be added to boost performance without increasing inference FLOPs. To address NPAS, we introduce Shapeshifter Networks (SSNs), which automatically learn where and how to share parameters in a network to support any parameter budget without requiring any changes to the architecture or loss function. NPAS and SSNs provide a complete framework for addressing generalized parameter sharing, and can also be combined with prior work for additional performance gains. We demonstrate the effectiveness of our approach using nine network architectures across four diverse tasks, including ImageNet classification and transformers.