# Controlling Model Complexity in Probabilistic Model-Based Dynamic Optimization of Neural Network Structures

arXiv: https://arxiv.org/abs/1907.06341 · HTML: https://mlsanity.com/paper/1907.06341/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jul 2019.

## Built on by (most cited first, 1 of 1)

- [Efficient Search of Multiple Neural Architectures with Different Complexities via Importance Sampling](https://mlsanity.com/paper/2207.10334/) (2022)

## Abstract (arXiv)

A method of simultaneously optimizing both the structure of neural networks and the connection weights in a single training loop can reduce the enormous computational cost of neural architecture search. We focus on the probabilistic model-based dynamic neural network structure optimization that considers the probability distribution of structure parameters and simultaneously optimizes both the distribution parameters and connection weights based on gradient methods. Since the existing algorithm searches for the structures that only minimize the training loss, this method might find overly complicated structures. In this paper, we propose the introduction of a penalty term to control the model complexity of obtained structures. We formulate a penalty term using the number of weights or units and derive its analytical natural gradient. The proposed method minimizes the objective function injected the penalty term based on the stochastic gradient descent. We apply the proposed method in the unit selection of a fully-connected neural network and the connection selection of a convolutional neural network. The experimental results show that the proposed method can control model complexity while maintaining performance.