# CONetV2: Efficient Auto-Channel Size Optimization for CNNs

arXiv: https://arxiv.org/abs/2110.06830 · HTML: https://mlsanity.com/paper/2110.06830/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 41 of its 48 references that are in the map.
- First posted on arXiv 13 Oct 2021.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 41)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [Fantastic Generalization Measures and Where to Find Them](https://mlsanity.com/paper/1912.02178/) (2019)
- [Progressive Neural Architecture Search](https://mlsanity.com/paper/1712.00559/) (2017)
- [A Comprehensive Overhaul of Feature Distillation](https://mlsanity.com/paper/1904.01866/) (2019)
- [Designing Neural Network Architectures using Reinforcement Learning](https://mlsanity.com/paper/1611.02167/) (2016)
- [Net2Net: Accelerating Learning via Knowledge Transfer](https://mlsanity.com/paper/1511.05641/) (2015)
- [Zero-Cost Proxies for Lightweight NAS](https://mlsanity.com/paper/2101.08134/) (2021)
- [Wider or Deeper: Revisiting the ResNet Model for Visual Recognition](https://mlsanity.com/paper/1611.10080/) (2016)
- [Auto-Keras: An Efficient Neural Architecture Search System](https://mlsanity.com/paper/1806.10282/) (2018)
- [Towards Understanding Knowledge Distillation](https://mlsanity.com/paper/2105.13093/) (2021)
- [Genetic CNN](https://mlsanity.com/paper/1703.01513/) (2017)
- [Resnet in Resnet: Generalizing Residual Architectures](https://mlsanity.com/paper/1603.08029/) (2016)
- [NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size](https://mlsanity.com/paper/2009.00437/) (2020)
- [Evolving Deep Neural Networks](https://mlsanity.com/paper/1703.00548/) (2017)
- [Grid Search, Random Search, Genetic Algorithm: A Big Comparison for NAS](https://mlsanity.com/paper/1912.06059/) (2019)
- [In Search of Robust Measures of Generalization](https://mlsanity.com/paper/2010.11924/) (2020)
- [A Survey on Neural Architecture Search](https://mlsanity.com/paper/1905.01392/) (2019)
- [Automated Machine Learning: From Principles to Practices](https://mlsanity.com/paper/1810.13306/) (2018)
- [Evolving Reinforcement Learning Algorithms](https://mlsanity.com/paper/2101.03958/) (2021)
- [Path-Level Network Transformation for Efficient Architecture Search](https://mlsanity.com/paper/1806.02639/) (2018)
- [Simple And Efficient Architecture Search for Convolutional Neural Networks](https://mlsanity.com/paper/1711.04528/) (2017)

## Built on by (most cited first, 4 of 4)

- [DataDAM: Efficient Dataset Distillation with Attention Matching](https://mlsanity.com/paper/2310.00093/) (2023)
- [CFDP: Common Frequency Domain Pruning](https://mlsanity.com/paper/2306.04147/) (2023)
- [ProbMCL: Simple Probabilistic Contrastive Learning for Multi-label Visual Classification](https://mlsanity.com/paper/2401.01448/) (2024)
- [ATOM: Attention Mixer for Efficient Dataset Distillation](https://mlsanity.com/paper/2405.01373/) (2024)

## Abstract (arXiv)

Neural Architecture Search (NAS) has been pivotal in finding optimal network configurations for Convolution Neural Networks (CNNs). While many methods explore NAS from a global search-space perspective, the employed optimization schemes typically require heavy computational resources. This work introduces a method that is efficient in computationally constrained environments by examining the micro-search space of channel size. In tackling channel-size optimization, we design an automated algorithm to extract the dependencies within different connected layers of the network. In addition, we introduce the idea of knowledge distillation, which enables preservation of trained weights, admist trials where the channel sizes are changing. Further, since the standard performance indicators (accuracy, loss) fail to capture the performance of individual network components (providing an overall network evaluation), we introduce a novel metric that highly correlates with test accuracy and enables analysis of individual network layers. Combining dependency extraction, metrics, and knowledge distillation, we introduce an efficient searching algorithm, with simulated annealing inspired stochasticity, and demonstrate its effectiveness in finding optimal architectures that outperform baselines by a large margin.