# Single-Nodal Spontaneous Symmetry Breaking in NLP Models

arXiv: https://arxiv.org/abs/2601.20582 · HTML: https://mlsanity.com/paper/2601.20582/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 42 references that are in the map.
- First posted on arXiv 28 Jan 2026.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [Revealing the Dark Secrets of BERT](https://mlsanity.com/paper/1908.08593/) (2019)
- [Towards a universal mechanism for successful deep learning](https://mlsanity.com/paper/2309.07537/) (2023)
- [Unified CNNs and transformers underlying learning mechanism reveals multi-head attention modus vivendi](https://mlsanity.com/paper/2501.12900/) (2025)
- [Low-latency vision transformers via large-scale multi-head attention](https://mlsanity.com/paper/2506.23832/) (2025)
- [Tiny language models](https://mlsanity.com/paper/2507.14871/) (2025)
- [Learning Mechanism Underlying NLP Pre-Training and Fine-Tuning](https://mlsanity.com/paper/2509.03407/) (2025)
- [Simultaneous multi-patch-clamp and extracellular-array recordings: Single neuron reflects network activity](https://mlsanity.com/paper/1707.06549/) (2017)
- [Self-attention vector output similarities reveal how machines pay attention](https://mlsanity.com/paper/2512.21956/) (2025)

## Abstract (arXiv)

Spontaneous symmetry breaking in statistical mechanics primarily occurs during phase transitions at the thermodynamic limit where the Hamiltonian preserves inversion symmetry, yet the low-temperature free energy exhibits reduced symmetry. Herein, we demonstrate the emergence of spontaneous symmetry breaking in natural language processing (NLP) models during both pre-training and fine-tuning, even under deterministic dynamics and within a finite training architecture. This phenomenon occurs at the level of individual attention heads and is scaled-down to its small subset of nodes and also valid at a single-nodal level, where nodes acquire the capacity to learn a limited set of tokens after pre-training or labels after fine-tuning for a specific classification task. As the number of nodes increases, a crossover in learning ability occurs, governed by the tradeoff between a decrease following random-guess among increased possible outputs, and enhancement following nodal cooperation, which exceeds the sum of individual nodal capabilities. In contrast to spin-glass systems, where a microscopic state of frozen spins cannot be directly linked to the free-energy minimization goal, each nodal function in this framework contributes explicitly to the global network task and can be upper-bounded using convex hull analysis. Results are demonstrated using BERT-6 architecture pre-trained on Wikipedia dataset and fine-tuned on the FewRel classification task.