# Dense Optimizer : An Information Entropy-Guided Structural Search Method for Dense-like Neural Network Design

arXiv: https://arxiv.org/abs/2410.07499 · HTML: https://mlsanity.com/paper/2410.07499/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 52 references that are in the map.
- First posted on arXiv 10 Oct 2024.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [Single Path One-Shot Neural Architecture Search with Uniform Sampling](https://mlsanity.com/paper/1904.00420/) (2019)
- [A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects](https://mlsanity.com/paper/2004.02806/) (2020)
- [Large-Scale Evolution of Image Classifiers](https://mlsanity.com/paper/1703.01041/) (2017)
- [SNAS: Stochastic Neural Architecture Search](https://mlsanity.com/paper/1812.09926/) (2018)
- [Neural Architecture Search without Training](https://mlsanity.com/paper/2006.04647/) (2020)
- [FractalNet: Ultra-Deep Neural Networks without Residuals](https://mlsanity.com/paper/1605.07648/) (2016)
- [BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search](https://mlsanity.com/paper/1910.11858/) (2019)
- [Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges](https://mlsanity.com/paper/2107.05847/) (2021)
- [Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction](https://mlsanity.com/paper/2006.08558/) (2020)
- [A Survey on Evolutionary Neural Architecture Search](https://mlsanity.com/paper/2008.10937/) (2020)
- [Genetic CNN](https://mlsanity.com/paper/1703.01513/) (2017)
- [Dual Path Networks](https://mlsanity.com/paper/1707.01629/) (2017)
- [Hyper-Parameter Optimization: A Review of Algorithms and Applications](https://mlsanity.com/paper/2003.05689/) (2020)
- [Convolutional Networks with Dense Connectivity](https://mlsanity.com/paper/2001.02394/) (2020)
- [ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction](https://mlsanity.com/paper/2105.10446/) (2021)
- [Very Deep Convolutional Neural Networks for Raw Waveforms](https://mlsanity.com/paper/1610.00087/) (2016)
- [ZARTS: On Zero-order Optimization for Neural Architecture Search](https://mlsanity.com/paper/2110.04743/) (2021)
- [Latency-Aware Neural Architecture Search with Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2106.11890/) (2021)
- [Convolutional Neural Networks with Alternately Updated Clique](https://mlsanity.com/paper/1802.10419/) (2018)
- [DeepMAD: Mathematical Architecture Design for Deep Convolutional Neural Network](https://mlsanity.com/paper/2303.02165/) (2023)
- [Deep Neural Network Hyperparameter Optimization with Orthogonal Array Tuning](https://mlsanity.com/paper/1907.13359/) (2019)
- [Bayesian Optimization and Deep Learning forsteering wheel angle prediction](https://mlsanity.com/paper/2110.13629/) (2021)

## Abstract (arXiv)

Dense Convolutional Network has been continuously refined to adopt a highly efficient and compact architecture, owing to its lightweight and efficient structure. However, the current Dense-like architectures are mainly designed manually, it becomes increasingly difficult to adjust the channels and reuse level based on past experience. As such, we propose an architecture search method called Dense Optimizer that can search high-performance dense-like network automatically. In Dense Optimizer, we view the dense network as a hierarchical information system, maximize the network's information entropy while constraining the distribution of the entropy across each stage via a power law, thereby constructing an optimization problem. We also propose a branch-and-bound optimization algorithm, tightly integrates power-law principle with search space scaling to solve the optimization problem efficiently. The superiority of Dense Optimizer has been validated on different computer vision benchmark datasets. Specifically, Dense Optimizer completes high-quality search but only costs 4 hours with one CPU. Our searched model DenseNet-OPT achieved a top 1 accuracy of 84.3% on CIFAR-100, which is 5.97% higher than the original one.