mlsanityData to 14 Oct 2026

SoftModel: A Neural Model That Grows Its Own Topology -- Governed Structural Growth for Continual In-Service Learning

arXiv 2608.16409 ↗First posted 17 Aug 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Today, a neural system is almost always used in two phases -- trained, then deployed -- and in that regime it freezes twice: training ends, and the topology itself was never a degree of freedom. We take the opposite premise as an axiom -- total plasticity: no part of a model, including its structure, is ever frozen -- and derive the governance a lifelong learner then requires. The design's target regime is continual, in-service learning: a long-lived model on a non-stationary stream, whose stability comes from governance rather than immobility and whose capacity follows demand. The result is a growable soft model: an algebra of structural operators (width, hierarchy, composition, input interface, grown cycles, attention heads), each exact at application, budgeted, and audited, with adoption decided solely by a held-out reality gate that treats parametric and structural change uniformly. A complete from-scratch system realizes the whole account; its factory surface is operated end-to-end by a production LLM. Two conclusions follow from the axiom by construction: stability under lifelong change becomes an audit property of the lifecycle, and structure that follows demand removes the silent cap a fixed topology places on later capability where the capacity floor binds. A third is measured: in the worlds where this was measured, the marginal value of new capacity was unobservable before adoption, so workable growth governance took its ex-post form. The same governance extends to evaluative signals, and the core method is evaluated on standard continual-learning benchmarks, where governed growth preserves the ability to keep learning along long task sequences. A pre-registered experimental program adjudicates the mechanism and value claims on the tested problems and reports its failures at full prominence; the map -- positive and negative -- is the contribution.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 142 references that are in the map. First posted on arXiv 17 Aug 2026.

Builds on

26 of 142 references.

References · 26

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  5. 2016Progressive Neural Networkscited by 1,361
  6. 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 867
  7. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 506
  8. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  9. 2025Scaling Latent Reasoning via Looped Language Modelscited by 165
  10. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 161
  11. 2014Freeze-Thaw Bayesian Optimizationcited by 90
  12. 2025Self-Adapting Language Modelscited by 84
14 more
  1. 2014How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagationcited by 79
  2. 2023Interpretable Machine Learning for Science with PySR and SymbolicRegression.jlcited by 54
  3. 2024Gymnasium: A Standard Interface for Reinforcement Learning Environmentscited by 46
  4. 2024On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encodingcited by 36
  5. 2023Masked Structural Growth for 2x Faster Language Model Pre-trainingcited by 33
  6. 2017Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networkscited by 19
  7. 2023Self-Expanding Neural Networkscited by 14
  8. 2023Composable Function-preserving Expansions for Transformer Architecturescited by 13
  9. 2025Entropy-Guided Attention for Private LLMscited by 9
  10. 2024What should a neuron aim for? Designing local objective functions based on information theorycited by 3
  11. 2025Multiscaled Multi-Head Attention-based Video Transformer Network for Hand Gesture Recognitioncited by 2
  12. 2026Prism Transformer: Progressive Head Schedules for Hierarchical Attention Processingcited by 2
  13. 2025Self-Motivated Growing Neural Network for Adaptive Architecture via Local Structural Plasticitycited by 1
  14. 2026Surgical Repair of Collapsed Attention Heads in ALiBi Transformerscited by 1

Built on by

Not cited yet.