# SoftModel: A Neural Model That Grows Its Own Topology -- Governed Structural Growth for Continual In-Service Learning

arXiv: https://arxiv.org/abs/2608.16409 · HTML: https://mlsanity.com/paper/2608.16409/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 142 references that are in the map.
- First posted on arXiv 17 Aug 2026.

## Builds on (most cited first, 26 of 26)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [AlphaEvolve: A coding agent for scientific and algorithmic discovery](https://mlsanity.com/paper/2506.13131/) (2025)
- [Adaptive Computation Time for Recurrent Neural Networks](https://mlsanity.com/paper/1603.08983/) (2016)
- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [Scaling Latent Reasoning via Looped Language Models](https://mlsanity.com/paper/2510.25741/) (2025)
- [Stabilizing Transformer Training by Preventing Attention Entropy Collapse](https://mlsanity.com/paper/2303.06296/) (2023)
- [Freeze-Thaw Bayesian Optimization](https://mlsanity.com/paper/1406.3896/) (2014)
- [Self-Adapting Language Models](https://mlsanity.com/paper/2506.10943/) (2025)
- [How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation](https://mlsanity.com/paper/1407.7906/) (2014)
- [Interpretable Machine Learning for Science with PySR and SymbolicRegression.jl](https://mlsanity.com/paper/2305.01582/) (2023)
- [Gymnasium: A Standard Interface for Reinforcement Learning Environments](https://mlsanity.com/paper/2407.17032/) (2024)
- [On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding](https://mlsanity.com/paper/2410.01405/) (2024)
- [Masked Structural Growth for 2x Faster Language Model Pre-training](https://mlsanity.com/paper/2305.02869/) (2023)
- [Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks](https://mlsanity.com/paper/1711.00350/) (2017)
- [Self-Expanding Neural Networks](https://mlsanity.com/paper/2307.04526/) (2023)
- [Composable Function-preserving Expansions for Transformer Architectures](https://mlsanity.com/paper/2308.06103/) (2023)
- [Entropy-Guided Attention for Private LLMs](https://mlsanity.com/paper/2501.03489/) (2025)
- [What should a neuron aim for? Designing local objective functions based on information theory](https://mlsanity.com/paper/2412.02482/) (2024)
- [Multiscaled Multi-Head Attention-based Video Transformer Network for Hand Gesture Recognition](https://mlsanity.com/paper/2501.00935/) (2025)
- [Prism Transformer: Progressive Head Schedules for Hierarchical Attention Processing](https://mlsanity.com/paper/2606.27449/) (2026)
- [Self-Motivated Growing Neural Network for Adaptive Architecture via Local Structural Plasticity](https://mlsanity.com/paper/2512.12713/) (2025)
- [Surgical Repair of Collapsed Attention Heads in ALiBi Transformers](https://mlsanity.com/paper/2603.09616/) (2026)

## Abstract (arXiv)

Today, a neural system is almost always used in two phases -- trained, then deployed -- and in that regime it freezes twice: training ends, and the topology itself was never a degree of freedom. We take the opposite premise as an axiom -- total plasticity: no part of a model, including its structure, is ever frozen -- and derive the governance a lifelong learner then requires. The design's target regime is continual, in-service learning: a long-lived model on a non-stationary stream, whose stability comes from governance rather than immobility and whose capacity follows demand. The result is a growable soft model: an algebra of structural operators (width, hierarchy, composition, input interface, grown cycles, attention heads), each exact at application, budgeted, and audited, with adoption decided solely by a held-out reality gate that treats parametric and structural change uniformly. A complete from-scratch system realizes the whole account; its factory surface is operated end-to-end by a production LLM. Two conclusions follow from the axiom by construction: stability under lifelong change becomes an audit property of the lifecycle, and structure that follows demand removes the silent cap a fixed topology places on later capability where the capacity floor binds. A third is measured: in the worlds where this was measured, the marginal value of new capacity was unobservable before adoption, so workable growth governance took its ex-post form. The same governance extends to evaluative signals, and the core method is evaluated on standard continual-learning benchmarks, where governed growth preserves the ability to keep learning along long task sequences. A pre-registered experimental program adjudicates the mechanism and value claims on the tested problems and reports its failures at full prominence; the map -- positive and negative -- is the contribution.