# Energy-Aware Neural Architecture Optimization with Fast Splitting Steepest Descent

arXiv: https://arxiv.org/abs/1910.03103 · HTML: https://mlsanity.com/paper/1910.03103/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Oct 2019.

## Built on by (most cited first, 18 of 18)

- [bert2BERT: Towards Reusable Pretrained Language Models](https://mlsanity.com/paper/2110.07143/) (2021)
- [GradMax: Growing Neural Networks using Gradient Information](https://mlsanity.com/paper/2201.05125/) (2022)
- [A Comprehensive Survey on Hardware-Aware Neural Architecture Search](https://mlsanity.com/paper/2101.09336/) (2021)
- [Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks](https://mlsanity.com/paper/2102.08574/) (2021)
- [Towards Green Automated Machine Learning: Status Quo and Future Directions](https://mlsanity.com/paper/2111.05850/) (2021)
- [Reusing Pretrained Models by Multi-linear Operators for Efficient Training](https://mlsanity.com/paper/2310.10699/) (2023)
- [Steepest Descent Density Control for Compact 3D Gaussian Splatting](https://mlsanity.com/paper/2505.05587/) (2025)
- [Preparing Lessons for Progressive Training on Language Models](https://mlsanity.com/paper/2401.09192/) (2024)
- [Split-Ensemble: Efficient OOD-aware Ensemble via Task and Model Splitting](https://mlsanity.com/paper/2312.09148/) (2023)
- [Energy-Aware Deep Learning on Resource-Constrained Hardware](https://mlsanity.com/paper/2505.12523/) (2025)
- [Learning Morphisms with Gauss-Newton Approximation for Growing Networks](https://mlsanity.com/paper/2411.05855/) (2024)
- [Saddle Hierarchy in Dense Associative Memory](https://mlsanity.com/paper/2508.19151/) (2025)
- [Shared-Weights Extender and Gradient Voting for Neural Network Expansion](https://mlsanity.com/paper/2509.18842/) (2025)
- [Growing Representation Learning](https://mlsanity.com/paper/2110.08857/) (2021)
- [Growing Neural Network with Shared Parameter](https://mlsanity.com/paper/2201.06500/) (2022)
- [Energy Concerns with HPC Systems and Applications](https://mlsanity.com/paper/2309.08615/) (2023)
- [Beyond Uniform Scaling: Exploring Depth Heterogeneity in Neural Architectures](https://mlsanity.com/paper/2402.12418/) (2024)
- [Explaining Machine Learning and Memorization with Statistical Mechanics](https://mlsanity.com/paper/2606.31110/) (2026)

## Abstract (arXiv)

Designing energy-efficient networks is of critical importance for enabling state-of-the-art deep learning in mobile and edge settings where the computation and energy budgets are highly limited. Recently, Liu et al. (2019) framed the search of efficient neural architectures into a continuous splitting process: it iteratively splits existing neurons into multiple off-springs to achieve progressive loss minimization, thus finding novel architectures by gradually growing the neural network. However, this method was not specifically tailored for designing energy-efficient networks, and is computationally expensive on large-scale benchmarks. In this work, we substantially improve Liu et al. (2019) in two significant ways: 1) we incorporate the energy cost of splitting different neurons to better guide the splitting process, thereby discovering more energy-efficient network architectures; 2) we substantially speed up the splitting process of Liu et al. (2019), which requires expensive eigen-decomposition, by proposing a highly scalable Rayleigh-quotient stochastic gradient algorithm. Our fast algorithm allows us to reduce the computational cost of splitting to the same level of typical back-propagation updates and enables efficient implementation on GPU. Extensive empirical results show that our method can train highly accurate and energy-efficient networks on challenging datasets such as ImageNet, improving a variety of baselines, including the pruning-based methods and expert-designed architectures.