# DC and SA: Robust and Efficient Hyperparameter Optimization of Multi-subnetwork Deep Learning Models

arXiv: https://arxiv.org/abs/2202.11841 · HTML: https://mlsanity.com/paper/2202.11841/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 34 references that are in the map.
- First posted on arXiv 24 Feb 2022.

## Builds on (most cited first, 12 of 12)

- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [Net2Net: Accelerating Learning via Knowledge Transfer](https://mlsanity.com/paper/1511.05641/) (2015)
- [MFAS: Multimodal Fusion Architecture Search](https://mlsanity.com/paper/1903.06496/) (2019)
- [A deep learning model integrating FCNNs and CRFs for brain tumor segmentation](https://mlsanity.com/paper/1702.04528/) (2017)
- [Deep Multimodal Neural Architecture Search](https://mlsanity.com/paper/2004.12070/) (2020)
- [Predicting drug response of tumors from integrated genomic profiles by deep neural networks](https://mlsanity.com/paper/1805.07702/) (2018)
- [Investigating EEG-Based Functional Connectivity Patterns for Multimodal Emotion Recognition](https://mlsanity.com/paper/2004.01973/) (2020)

## Abstract (arXiv)

We present two novel hyperparameter optimization strategies for optimization of deep learning models with a modular architecture constructed of multiple subnetworks. As complex networks with multiple subnetworks become more frequently applied in machine learning, hyperparameter optimization methods are required to efficiently optimize their hyperparameters. Existing hyperparameter searches are general, and can be used to optimize such networks, however, by exploiting the multi-subnetwork architecture, these searches can be sped up substantially. The proposed methods offer faster convergence to a better-performing final model. To demonstrate this, we propose 2 independent approaches to enhance these prior algorithms: 1) a divide-and-conquer approach, in which the best subnetworks of top-performing models are combined, allowing for more rapid sampling of the hyperparameter search space. 2) A subnetwork adaptive approach that distributes computational resources based on the importance of each subnetwork, allowing more intelligent resource allocation. These approaches can be flexibily applied to many hyperparameter optimization algorithms. To illustrate this, we combine our approaches with the commonly-used Bayesian optimization method. Our approaches are then tested against both synthetic examples and real-world examples and applied to multiple network types including convolutional neural networks and dense feed forward neural networks. Our approaches show an increased optimization efficiency of up to 23.62x, and a final performance boost of up to 3.5% accuracy for classification and 4.4 MSE for regression, when compared to comparable BO approach.