# Four Ways to Grow a Classifier and Why One of Them Cannot Learn

arXiv: https://arxiv.org/abs/2610.00180 · HTML: https://mlsanity.com/paper/2610.00180/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 14 references that are in the map.
- First posted on arXiv 16 Sep 2026.

## Builds on (most cited first, 7 of 7)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Net2Net: Accelerating Learning via Knowledge Transfer](https://mlsanity.com/paper/1511.05641/) (2015)
- [Distilling a Neural Network Into a Soft Decision Tree](https://mlsanity.com/paper/1711.09784/) (2017)
- [GradMax: Growing Neural Networks using Gradient Information](https://mlsanity.com/paper/2201.05125/) (2022)
- [Network Morphism](https://mlsanity.com/paper/1603.01670/) (2016)
- [Splitting Steepest Descent for Growing Neural Architectures](https://mlsanity.com/paper/1910.02366/) (2019)
- [Dropout Regularization in Hierarchical Mixture of Experts](https://mlsanity.com/paper/1812.10158/) (2018)

## Abstract (arXiv)

Constructive classifiers add structure while they train: a level to a tree, a unit to a hidden layer, a split at a leaf. This paper asks what each of four such growth decisions buys, measured under one protocol on 24 datasets, and gives an exact diagnosis and fix for the one that buys nothing. The diagnosis concerns the natural way to deepen a soft decision tree: turn every leaf into a gate whose two children inherit the parent's class distribution, so the function is unchanged. I prove that this leaves the gradient of every new gate identically zero and with the gate at 1/2, gives the two children identical gradients, so under this construction the added level can never learn. That predicts where it costs: nothing on two-class problems, where two leaves already suffice and a great deal where more classes need more leaves. Measured, the cost is -0.2 points over 13 binary datasets and 40.2 points over 7 multi-class ones, and it tracks the number of classes (Spearman 0.64), not the number of features (0.03). The fix is any perturbation of the children. Neither its size nor its direction matters: a residual-directed initialisation changes accuracy by +0.20 points against noise. The other three decisions each buy one thing. Fitting a new hidden unit to the residual before installing it buys a smaller network but not a more accurate one. Splitting one leaf at a time buys sparsity; it lost accuracy until I found that the split started its two children identical and untrained, the same defect in another place; with the children inheriting the parent and the symmetry broken, per-leaf growth comes within 2.3 points of a complete tree using 23% of its splits. Requiring statistical significance before a node gets a more expressive split buys nothing. Every number comes from the measurement scripts.