# Abrupt and spontaneous strategy switches emerge in simple regularised neural networks

arXiv: https://arxiv.org/abs/2302.11351 · HTML: https://mlsanity.com/paper/2302.11351/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 73 references that are in the map.
- First posted on arXiv 22 Feb 2023.

## Builds on (most cited first, 8 of 8)

- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [A mathematical theory of semantic development in deep neural networks](https://mlsanity.com/paper/1810.10531/) (2018)
- [Bad Global Minima Exist and SGD Can Reach Them](https://mlsanity.com/paper/1906.02613/) (2019)
- [Theory of gating in recurrent neural networks](https://mlsanity.com/paper/2007.14823/) (2020)
- [The Overfitted Brain: Dreams evolved to assist generalization](https://mlsanity.com/paper/2007.09560/) (2020)
- [Cortical microcircuits as gated-recurrent neural networks](https://mlsanity.com/paper/1711.02448/) (2017)
- [Cognitive control as a multivariate optimization problem](https://mlsanity.com/paper/2110.00668/) (2021)

## Built on by (most cited first, 2 of 2)

- [Random Scaling of Emergent Capabilities](https://mlsanity.com/paper/2502.17356/) (2025)
- [On the Emergence of Induction Heads for In-Context Learning](https://mlsanity.com/paper/2511.01033/) (2025)

## Abstract (arXiv)

Humans sometimes have an insight that leads to a sudden and drastic performance improvement on the task they are working on. Sudden strategy adaptations are often linked to insights, considered to be a unique aspect of human cognition tied to complex processes such as creativity or meta-cognitive reasoning. Here, we take a learning perspective and ask whether insight-like behaviour can occur in simple artificial neural networks, even when the models only learn to form input-output associations through gradual gradient descent. We compared learning dynamics in humans and regularised neural networks in a perceptual decision task that included a hidden regularity to solve the task more efficiently. Our results show that only some humans discover this regularity, whose behaviour was marked by a sudden and abrupt strategy switch that reflects an aha-moment. Notably, we find that simple neural networks with a gradual learning rule and a constant learning rate closely mimicked behavioural characteristics of human insight-like switches, exhibiting delay of insight, suddenness and selective occurrence in only some networks. Analyses of network architectures and learning dynamics revealed that insight-like behaviour crucially depended on a regularised gating mechanism and noise added to gradient updates, which allowed the networks to accumulate "silent knowledge" that is initially suppressed by regularised (attentional) gating. This suggests that insight-like behaviour can arise naturally from gradual learning in simple neural networks, where it reflects the combined influences of noise, gating and regularisation.