# Robustness of the Random Language Model

arXiv: https://arxiv.org/abs/2309.14913 · HTML: https://mlsanity.com/paper/2309.14913/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 21 references that are in the map.
- First posted on arXiv 26 Sep 2023.

## Builds on (most cited first, 5 of 5)

- [A mathematical theory of semantic development in deep neural networks](https://mlsanity.com/paper/1810.10531/) (2018)
- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Scale-free networks are rare](https://mlsanity.com/paper/1801.03400/) (2018)
- [Recurrent Neural Networks in Linguistic Theory: Revisiting Pinker and Prince (1988) and the Past Tense Debate](https://mlsanity.com/paper/1807.04783/) (2018)
- [Emergence of order in random languages](https://mlsanity.com/paper/1902.07516/) (2019)

## Built on by (most cited first, 1 of 1)

- [How transformers learn structured data: insights from hierarchical filtering](https://mlsanity.com/paper/2408.15138/) (2024)

## Abstract (arXiv)

The Random Language Model (De Giuli 2019) is an ensemble of stochastic context-free grammars, quantifying the syntax of human and computer languages. The model suggests a simple picture of first language learning as a type of annealing in the vast space of potential languages. In its simplest formulation, it implies a single continuous transition to grammatical syntax, at which the symmetry among potential words and categories is spontaneously broken. Here this picture is scrutinized by considering its robustness against extensions of the original model, and trajectories through parameter space different from those originally considered. It is shown here that (i) the scenario is robust to explicit symmetry breaking, an inevitable component of learning in the real world; and (ii) the transition to grammatical syntax can be encountered by fixing the deep (hidden) structure while varying the surface (observable) properties. It is also argued that the transition becomes a sharp thermodynamic transition in an idealized limit. Moreover, comparison with human data on the clustering coefficient of syntax networks suggests that the observed transition is equivalent to that normally experienced by children at age 24 months. The results are discussed in light of theory of first-language acquisition in linguistics, and recent successes in machine learning.