# Learning Neural Network Architectures using Backpropagation

arXiv: https://arxiv.org/abs/1511.05497 · HTML: https://mlsanity.com/paper/1511.05497/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Nov 2015.

## Built on by (most cited first, 5 of 5)

- [Sparsified Model Zoo Twins: Investigating Populations of Sparsified Neural Network Models](https://mlsanity.com/paper/2304.13718/) (2023)
- [Towards Self-Adaptive Metric Learning On the Fly](https://mlsanity.com/paper/2104.01495/) (2021)
- [Exact Backpropagation in Binary Weighted Networks with Group Weight Transformations](https://mlsanity.com/paper/2107.01400/) (2021)
- [Learning a Consensus Sub-Network with Polarization Regularization and One Pass Training](https://mlsanity.com/paper/2302.10798/) (2023)
- [Soft Diamond Regularizers for Deep Learning](https://mlsanity.com/paper/2412.20724/) (2024)

## Abstract (arXiv)

Deep neural networks with millions of parameters are at the heart of many state of the art machine learning models today. However, recent works have shown that models with much smaller number of parameters can also perform just as well. In this work, we introduce the problem of architecture-learning, i.e; learning the architecture of a neural network along with weights. We introduce a new trainable parameter called tri-state ReLU, which helps in eliminating unnecessary neurons. We also propose a smooth regularizer which encourages the total number of neurons after elimination to be small. The resulting objective is differentiable and simple to optimize. We experimentally validate our method on both small and large networks, and show that it can learn models with a considerably small number of parameters without affecting prediction accuracy.