# CAGE-NAS: Certified Functional Descent for Efficient Model Growth

arXiv: https://arxiv.org/abs/2610.01173 · HTML: https://mlsanity.com/paper/2610.01173/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 11 references that are in the map.
- First posted on arXiv 1 Oct 2026.

## Builds on (most cited first, 8 of 8)

- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [GradMax: Growing Neural Networks using Gradient Information](https://mlsanity.com/paper/2201.05125/) (2022)
- [Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks](https://mlsanity.com/paper/2102.08574/) (2021)
- [When, where, and how to add new neurons to ANNs](https://mlsanity.com/paper/2202.08539/) (2022)
- [Growing Tiny Networks: Spotting Expressivity Bottlenecks and Fixing Them Optimally](https://mlsanity.com/paper/2405.19816/) (2024)
- [When To Grow? A Fitting Risk-Aware Policy for Layer Growing in Deep Neural Networks](https://mlsanity.com/paper/2401.03104/) (2024)
- [Growth strategies for arbitrary DAG neural architectures](https://mlsanity.com/paper/2501.12690/) (2025)
- [Functional Gradient Descent with Adaptive Representations](https://mlsanity.com/paper/2606.16926/) (2026)

## Abstract (arXiv)

The progressive growth of neural networks requires deciding when the current representation remains sufficient for optimization and when it should be expanded. CAGE-NAS formulates this decision in function space through an admissibility criterion on approximations of the functional gradient. As long as a representation enables a certified Functional Gradient Descent step, the architecture remains fixed; when the criterion fails, a function-preserving expansion is applied and the resulting representation is evaluated again. As the main instance, we study the family induced by the tangent space, using a regularized projection of the functional gradient. In a controlled setting with exact certification, CAGE-NAS produces architectures positioned above the 99.8th performance percentile by held-out RMSE among all admissible alternatives within the same parameter budget, without enumerating them during the growth trajectory.