# Shared-Weights Extender and Gradient Voting for Neural Network Expansion

arXiv: https://arxiv.org/abs/2509.18842 · HTML: https://mlsanity.com/paper/2509.18842/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 20 references that are in the map.
- First posted on arXiv 23 Sep 2025.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [Net2Net: Accelerating Learning via Knowledge Transfer](https://mlsanity.com/paper/1511.05641/) (2015)
- [Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search](https://mlsanity.com/paper/1911.12126/) (2019)
- [GradMax: Growing Neural Networks using Gradient Information](https://mlsanity.com/paper/2201.05125/) (2022)
- [Splitting Steepest Descent for Growing Neural Architectures](https://mlsanity.com/paper/1910.02366/) (2019)
- [Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks](https://mlsanity.com/paper/2102.08574/) (2021)
- [Energy-Aware Neural Architecture Optimization with Fast Splitting Steepest Descent](https://mlsanity.com/paper/1910.03103/) (2019)
- [Self-Expanding Neural Networks](https://mlsanity.com/paper/2307.04526/) (2023)

## Built on by (most cited first, 1 of 1)

- [Preservation Is Not Enough for Width Growth: Regime-Sensitive Selection of Dense LM Warm Starts](https://mlsanity.com/paper/2604.04281/) (2026)

## Abstract (arXiv)

Expanding neural networks during training is a promising way to augment capacity without retraining larger models from scratch. However, newly added neurons often fail to adjust to a trained network and become inactive, providing no contribution to capacity growth. We propose the Shared-Weights Extender (SWE), a novel method explicitly designed to prevent inactivity of new neurons by coupling them with existing ones for smooth integration. In parallel, we introduce the Steepest Voting Distributor (SVoD), a gradient-based method for allocating neurons across layers during deep network expansion. Our extensive benchmarking on four datasets shows that our method can effectively suppress neuron inactivity and achieve better performance compared to other expanding methods and baselines.