# Intelligence at the Edge of Chaos

arXiv: https://arxiv.org/abs/2410.02536 · HTML: https://mlsanity.com/paper/2410.02536/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 56 references that are in the map.
- First posted on arXiv 3 Oct 2024.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning](https://mlsanity.com/paper/2312.15685/) (2023)
- [Selection via Proxy: Efficient Data Selection for Deep Learning](https://mlsanity.com/paper/1906.11829/) (2019)
- [A Preliminary Study of the Intrinsic Relationship between Complexity and Alignment](https://mlsanity.com/paper/2308.05696/) (2023)
- [Criticality in the brain: A synthesis of neurobiology, models and cognition](https://mlsanity.com/paper/1707.05952/) (2017)
- [LifeGPT: Topology-Agnostic Generative Pretrained Transformer Model for Cellular Automata](https://mlsanity.com/paper/2409.12182/) (2024)

## Built on by (most cited first, 13 of 13)

- [Training Language Models via Neural Cellular Automata](https://mlsanity.com/paper/2603.10055/) (2026)
- [Procedural Pretraining: Warming Up Language Models with Abstract Data](https://mlsanity.com/paper/2601.21725/) (2026)
- [Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning](https://mlsanity.com/paper/2505.22308/) (2025)
- [Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformers](https://mlsanity.com/paper/2511.13945/) (2025)
- [Advancing the Scientific Method with Large Language Models: From Hypothesis to Discovery](https://mlsanity.com/paper/2505.16477/) (2025)
- [Cognitive Activation and Chaotic Dynamics in Large Language Models: A Quasi-Lyapunov Analysis of Reasoning Mechanisms](https://mlsanity.com/paper/2503.13530/) (2025)
- [Small Models, Smarter Learning: The Power of Joint Task Training](https://mlsanity.com/paper/2505.18369/) (2025)
- [Intuition emerges in Maximum Caliber models at criticality](https://mlsanity.com/paper/2508.06477/) (2025)
- [Complexity-Regularized Proximal Policy Optimization](https://mlsanity.com/paper/2509.20509/) (2025)
- [Evolving Many Worlds: Towards Open-Ended Discovery in Petri Dish NCA via Population-Based Training](https://mlsanity.com/paper/2604.11248/) (2026)
- [Procedural Core: A Compact Recurrent Initialization for Vision Transformers](https://mlsanity.com/paper/2609.37631/) (2026)
- [Lasting Effects of Abstract Pretraining Beyond Perplexity](https://mlsanity.com/paper/2609.38764/) (2026)
- [Bellman Meets Lyapunov: Unsupervised Reinforcement Learning via Mastering Chaos](https://mlsanity.com/paper/2610.02012/) (2026)

## Abstract (arXiv)

We explore the emergence of intelligent behavior in artificial systems by investigating how the complexity of rule-based systems influences the capabilities of models trained to predict these rules. Our study focuses on elementary cellular automata (ECA), simple yet powerful one-dimensional systems that generate behaviors ranging from trivial to highly complex. By training distinct Large Language Models (LLMs) on different ECAs, we evaluated the relationship between the complexity of the rules' behavior and the intelligence exhibited by the LLMs, as reflected in their performance on downstream tasks. Our findings reveal that rules with higher complexity lead to models exhibiting greater intelligence, as demonstrated by their performance on reasoning and chess move prediction tasks. Both uniform and periodic systems, and often also highly chaotic systems, resulted in poorer downstream performance, highlighting a sweet spot of complexity conducive to intelligence. We conjecture that intelligence arises from the ability to predict complexity and that creating intelligence may require only exposure to complexity.