# Over the Edge of Chaos? Excess Complexity as a Roadblock to Artificial General Intelligence

arXiv: https://arxiv.org/abs/2407.03652 · HTML: https://mlsanity.com/paper/2407.03652/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 73 references that are in the map.
- First posted on arXiv 4 Jul 2024.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023)
- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [Augmented Language Models: a Survey](https://mlsanity.com/paper/2302.07842/) (2023)
- [DayDreamer: World Models for Physical Robot Learning](https://mlsanity.com/paper/2206.14176/) (2022)
- [Evaluating Large Language Models in Theory of Mind Tasks](https://mlsanity.com/paper/2302.02083/) (2023)
- [Liquid Time-constant Networks](https://mlsanity.com/paper/2006.04439/) (2020)
- [The Generative AI Paradox: "What It Can Create, It May Not Understand"](https://mlsanity.com/paper/2311.00059/) (2023)
- [Can Large Language Models Reason and Plan?](https://mlsanity.com/paper/2403.04121/) (2024)
- [No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance](https://mlsanity.com/paper/2404.04125/) (2024)
- [Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence](https://mlsanity.com/paper/2402.09880/) (2024)
- [Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models](https://mlsanity.com/paper/2406.02061/) (2024)
- [When Brain-inspired AI Meets AGI](https://mlsanity.com/paper/2303.15935/) (2023)
- [Grokking as a First Order Phase Transition in Two Layer Networks](https://mlsanity.com/paper/2310.03789/) (2023)
- [Brain-inspired learning in artificial neural networks: a review](https://mlsanity.com/paper/2305.11252/) (2023)
- [Human-like object concept representations emerge naturally in multimodal large language models](https://mlsanity.com/paper/2407.01067/) (2024)
- [AI Autonomy : Self-Initiated Open-World Continual Learning and Adaptation](https://mlsanity.com/paper/2203.08994/) (2022)
- [From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscape](https://mlsanity.com/paper/2312.10868/) (2023)
- [The Generative AI Paradox on Evaluation: What It Can Solve, It May Not Evaluate](https://mlsanity.com/paper/2402.06204/) (2024)
- [Superintelligence cannot be contained: Lessons from Computability Theory](https://mlsanity.com/paper/1607.00913/) (2016)
- [Optimal Machine Intelligence at the Edge of Chaos](https://mlsanity.com/paper/1909.05176/) (2019)
- [Self-Organized Criticality in the Brain](https://mlsanity.com/paper/2102.09124/) (2021)
- [Optimal input representation in neural systems at the edge of chaos](https://mlsanity.com/paper/2107.05709/) (2021)

## Built on by (most cited first, 2 of 2)

- [Cognitive Activation and Chaotic Dynamics in Large Language Models: A Quasi-Lyapunov Analysis of Reasoning Mechanisms](https://mlsanity.com/paper/2503.13530/) (2025)
- [A second-order-like optimizer with adaptive gradient scaling for deep learning](https://mlsanity.com/paper/2410.05871/) (2024)

## Abstract (arXiv)

In this study, we explored the progression trajectories of artificial intelligence (AI) systems through the lens of complexity theory. We challenged the conventional linear and exponential projections of AI advancement toward Artificial General Intelligence (AGI) underpinned by transformer-based architectures, and posited the existence of critical points, akin to phase transitions in complex systems, where AI performance might plateau or regress into instability upon exceeding a critical complexity threshold. We employed agent-based modelling (ABM) to simulate hypothetical scenarios of AI systems' evolution under specific assumptions, using benchmark performance as a proxy for capability and complexity. Our simulations demonstrated how increasing the complexity of the AI system could exceed an upper criticality threshold, leading to unpredictable performance behaviours. Additionally, we developed a practical methodology for detecting these critical thresholds using simulation data and stochastic gradient descent to fine-tune detection thresholds. This research offers a novel perspective on AI advancement that has a particular relevance to Large Language Models (LLMs), emphasising the need for a tempered approach to extrapolating AI's growth potential and underscoring the importance of developing more robust and comprehensive AI performance benchmarks.