# Learning Elementary Cellular Automata with Transformers

arXiv: https://arxiv.org/abs/2412.01417 · HTML: https://mlsanity.com/paper/2412.01417/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 2 Dec 2024.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018)
- [Faith and Fate: Limits of Transformers on Compositionality](https://mlsanity.com/paper/2305.18654/) (2023)
- [Are Transformers universal approximators of sequence-to-sequence functions?](https://mlsanity.com/paper/1912.10077/) (2019)
- [Deep Learning for Symbolic Mathematics](https://mlsanity.com/paper/1912.01412/) (2019)
- [End-to-end symbolic regression with transformers](https://mlsanity.com/paper/2204.10532/) (2022)
- [Representational Strengths and Limitations of Transformers](https://mlsanity.com/paper/2306.02896/) (2023)
- [On the Computational Power of Transformers and its Implications in Sequence Modeling](https://mlsanity.com/paper/2006.09286/) (2020)
- [Deep Symbolic Regression for Recurrent Sequences](https://mlsanity.com/paper/2201.04600/) (2022)
- [Cellular automata as convolutional neural networks](https://mlsanity.com/paper/1809.02942/) (2018)
- [Learning Graph Cellular Automata](https://mlsanity.com/paper/2110.14237/) (2021)
- [Attention-based Neural Cellular Automata](https://mlsanity.com/paper/2211.01233/) (2022)
- [It's Hard for Neural Networks To Learn the Game of Life](https://mlsanity.com/paper/2009.01398/) (2020)
- [E(n)-equivariant Graph Neural Cellular Automata](https://mlsanity.com/paper/2301.10497/) (2023)
- [Conditional and Modal Reasoning in Large Language Models](https://mlsanity.com/paper/2401.17169/) (2024)
- [Liar, Liar, Logical Mire: A Benchmark for Suppositional Reasoning in Large Language Models](https://mlsanity.com/paper/2406.12546/) (2024)
- [Learning spatio-temporal patterns with Neural Cellular Automata](https://mlsanity.com/paper/2310.14809/) (2023)
- [LifeGPT: Topology-Agnostic Generative Pretrained Transformer Model for Cellular Automata](https://mlsanity.com/paper/2409.12182/) (2024)
- [Generalization over different cellular automata rules learned by a deep feed-forward neural network](https://mlsanity.com/paper/2103.14886/) (2021)
- [Reconstructing cellular automata rules from observations at nonconsecutive times](https://mlsanity.com/paper/2012.02179/) (2020)
- [Learning Locally Interacting Discrete Dynamical Systems: Towards Data-Efficient and Scalable Prediction](https://mlsanity.com/paper/2404.06460/) (2024)
- [Cellular automata, many-valued logic, and deep neural networks](https://mlsanity.com/paper/2404.05259/) (2024)
- [Data-Centric Approach to Constrained Machine Learning: A Case Study on Conway's Game of Life](https://mlsanity.com/paper/2408.12778/) (2024)

## Built on by (most cited first, 4 of 4)

- [ARC-NCA: Towards Developmental Solutions to the Abstraction and Reasoning Corpus](https://mlsanity.com/paper/2505.08778/) (2025)
- [AutomataGPT: Forecasting and Ruleset Inference for Two-Dimensional Cellular Automata](https://mlsanity.com/paper/2506.17333/) (2025)
- [Why Do Conventional World Models Fail to Learn Cellular Automata?](https://mlsanity.com/paper/2609.39604/) (2026)
- [Spatial Induction Heads: In-Context Learning of Multidimensional Cellular Automata](https://mlsanity.com/paper/2610.09124/) (2026)

## Abstract (arXiv)

Large Language Models demonstrate remarkable mathematical capabilities but at the same time struggle with abstract reasoning and planning. In this study, we explore whether Transformers can learn to abstract and generalize the rules governing Elementary Cellular Automata. By training Transformers on state sequences generated with random initial conditions and local rules, we show that they can generalize across different Boolean functions of fixed arity, effectively abstracting the underlying rules. While the models achieve high accuracy in next-state prediction, their performance declines sharply in multi-step planning tasks without intermediate context. Our analysis reveals that including future states or rule prediction in the training loss enhances the models' ability to form internal representations of the rules, leading to improved performance in longer planning horizons and autoregressive generation. Furthermore, we confirm that increasing the model's depth plays a crucial role in extended sequential computations required for complex reasoning tasks. This highlights the potential to improve LLM with inclusion of longer horizons in loss function, as well as incorporating recurrence and adaptive computation time for dynamic control of model depth.