# Steepest Descent Neural Architecture Optimization: Escaping Local Optimum with Signed Neural Splitting

arXiv: https://arxiv.org/abs/2003.10392 · HTML: https://mlsanity.com/paper/2003.10392/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Mar 2020.

## Built on by (most cited first, 14 of 14)

- [bert2BERT: Towards Reusable Pretrained Language Models](https://mlsanity.com/paper/2110.07143/) (2021)
- [GradMax: Growing Neural Networks using Gradient Information](https://mlsanity.com/paper/2201.05125/) (2022)
- [Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks](https://mlsanity.com/paper/2102.08574/) (2021)
- [Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancement](https://mlsanity.com/paper/2409.19732/) (2024)
- [Sparsity Winning Twice: Better Robust Generalization from More Efficient Training](https://mlsanity.com/paper/2202.09844/) (2022)
- [Reusing Pretrained Models by Multi-linear Operators for Efficient Training](https://mlsanity.com/paper/2310.10699/) (2023)
- [Steepest Descent Density Control for Compact 3D Gaussian Splatting](https://mlsanity.com/paper/2505.05587/) (2025)
- [Preparing Lessons for Progressive Training on Language Models](https://mlsanity.com/paper/2401.09192/) (2024)
- [Accelerated Training via Incrementally Growing Neural Networks using Variance Transfer and Learning Rate Adaptation](https://mlsanity.com/paper/2306.12700/) (2023)
- [Split-Ensemble: Efficient OOD-aware Ensemble via Task and Model Splitting](https://mlsanity.com/paper/2312.09148/) (2023)
- [When To Grow? A Fitting Risk-Aware Policy for Layer Growing in Deep Neural Networks](https://mlsanity.com/paper/2401.03104/) (2024)
- [Flat Channels to Infinity in Neural Loss Landscapes](https://mlsanity.com/paper/2506.14951/) (2025)
- [Learning Morphisms with Gauss-Newton Approximation for Growing Networks](https://mlsanity.com/paper/2411.05855/) (2024)
- [Beyond Uniform Scaling: Exploring Depth Heterogeneity in Neural Architectures](https://mlsanity.com/paper/2402.12418/) (2024)

## Abstract (arXiv)

Developing efficient and principled neural architecture optimization methods is a critical challenge of modern deep learning. Recently, Liu et al.[19] proposed a splitting steepest descent (S2D) method that jointly optimizes the neural parameters and architectures based on progressively growing network structures by splitting neurons into multiple copies in a steepest descent fashion. However, S2D suffers from a local optimality issue when all the neurons become "splitting stable", a concept akin to local stability in parametric optimization. In this work, we develop a significant and surprising extension of the splitting descent framework that addresses the local optimality issue. The idea is to observe that the original S2D is unnecessarily restricted to splitting neurons into positive weighted copies. By simply allowing both positive and negative weights during splitting, we can eliminate the appearance of splitting stability in S2D and hence escape the local optima to obtain better performance. By incorporating signed splittings, we significantly extend the optimization power of splitting steepest descent both theoretically and empirically. We verify our method on various challenging benchmarks such as CIFAR-100, ImageNet and ModelNet40, on which we outperform S2D and other advanced methods on learning accurate and energy-efficient neural networks.