# Attention-based Neural Cellular Automata

arXiv: https://arxiv.org/abs/2211.01233 · HTML: https://mlsanity.com/paper/2211.01233/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 72 references that are in the map.
- First posted on arXiv 2 Nov 2022.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [Masked Feature Prediction for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2112.09133/) (2021)
- [Noise2Noise: Learning Image Restoration without Clean Data](https://mlsanity.com/paper/1803.04189/) (2018)
- [XCiT: Cross-Covariance Image Transformers](https://mlsanity.com/paper/2106.09681/) (2021)
- [On the Relationship between Self-Attention and Convolutional Layers](https://mlsanity.com/paper/1911.03584/) (2019)
- [Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding](https://mlsanity.com/paper/2103.15358/) (2021)
- [On the Turing Completeness of Modern Neural Network Architectures](https://mlsanity.com/paper/1901.03429/) (2019)
- [RegionViT: Regional-to-Local Attention for Vision Transformers](https://mlsanity.com/paper/2106.02689/) (2021)
- [Generative Adversarial Transformers](https://mlsanity.com/paper/2103.01209/) (2021)
- [Deep Equilibrium Optical Flow Estimation](https://mlsanity.com/paper/2204.08442/) (2022)

## Built on by (most cited first, 26 of 26)

- [ARC-NCA: Towards Developmental Solutions to the Abstraction and Reasoning Corpus](https://mlsanity.com/paper/2505.08778/) (2025)
- [E(n)-equivariant Graph Neural Cellular Automata](https://mlsanity.com/paper/2301.10497/) (2023)
- [Arbitrarily Scalable Environment Generators via Neural Cellular Automata](https://mlsanity.com/paper/2310.18622/) (2023)
- [Differentiable Logic Cellular Automata: From Game of Life to Pattern Generation](https://mlsanity.com/paper/2506.04912/) (2025)
- [Neural cellular automata: applications to biology and beyond classical AI](https://mlsanity.com/paper/2509.11131/) (2025)
- [CAX: Cellular Automata Accelerated in JAX](https://mlsanity.com/paper/2410.02651/) (2024)
- [NoiseNCA: Noisy Seed Improves Spatio-Temporal Continuity of Neural Cellular Automata](https://mlsanity.com/paper/2404.06279/) (2024)
- [EngramNCA: a Neural Cellular Automaton Model of Memory Transfer](https://mlsanity.com/paper/2504.11855/) (2025)
- [Learning Locally Interacting Discrete Dynamical Systems: Towards Data-Efficient and Scalable Prediction](https://mlsanity.com/paper/2404.06460/) (2024)
- [Neural Cellular Automata: From Cells to Pixels](https://mlsanity.com/paper/2506.22899/) (2025)
- [Latent Neural Cellular Automata for Resource-Efficient Image Restoration](https://mlsanity.com/paper/2403.15525/) (2024)
- [Neural Cellular Automata for Lightweight, Robust and Explainable Classification of White Blood Cell Images](https://mlsanity.com/paper/2404.05584/) (2024)
- [AdaNCA: Neural Cellular Automata As Adaptors For More Robust Vision Transformer](https://mlsanity.com/paper/2406.08298/) (2024)
- [An Organism Starts with a Single Pix-Cell: A Neural Cellular Diffusion for High-Resolution Image Synthesis](https://mlsanity.com/paper/2407.03018/) (2024)
- [Learning Elementary Cellular Automata with Transformers](https://mlsanity.com/paper/2412.01417/) (2024)
- [Beyond Memorization: Extending Reasoning Depth with Recurrence, Memory and Test-Time Compute Scaling](https://mlsanity.com/paper/2508.16745/) (2025)
- [AutomataGPT: Forecasting and Ruleset Inference for Two-Dimensional Cellular Automata](https://mlsanity.com/paper/2506.17333/) (2025)
- [A New Kind of Network? Review and Reference Implementation of Neural Cellular Automata](https://mlsanity.com/paper/2604.24990/) (2026)
- [MedSegDiffNCA: Diffusion Models With Neural Cellular Automata for Skin Lesion Segmentation](https://mlsanity.com/paper/2501.02447/) (2025)
- [Attention Pooling Enhances NCA-based Classification of Microscopy Images](https://mlsanity.com/paper/2508.12324/) (2025)
- [On the Emergence of Syntax by Means of Local Interaction](https://mlsanity.com/paper/2604.17857/) (2026)
- [Mixtures of Neural Cellular Automata: A Stochastic Framework for Growth Modelling and Self-Organization](https://mlsanity.com/paper/2506.20486/) (2025)
- [Missing Data Imputation using Neural Cellular Automata](https://mlsanity.com/paper/2509.00651/) (2025)
- [DiffeoMorph: Learning to Morph 3D Shapes Using Differentiable Agent-Based Simulations](https://mlsanity.com/paper/2512.17129/) (2025)
- [Applications of Neural Cellular Automata: State of the Art, Challenges and Opportunities](https://mlsanity.com/paper/2609.24595/) (2026)
- [Where Does Randomness Matter in Neural Cellular Automata?](https://mlsanity.com/paper/2609.36797/) (2026)

## Abstract (arXiv)

Recent extensions of Cellular Automata (CA) have incorporated key ideas from modern deep learning, dramatically extending their capabilities and catalyzing a new family of Neural Cellular Automata (NCA) techniques. Inspired by Transformer-based architectures, our work presents a new class of textitattention-based NCAs formed using a spatially localizedunicodex2014yet globally organizedunicodex2014self-attention scheme. We introduce an instance of this class named textitVision Transformer Cellular Automata (ViTCA). We present quantitative and qualitative results on denoising autoencoding across six benchmark datasets, comparing ViTCA to a U-Net, a U-Net-based CA baseline (UNetCA), and a Vision Transformer (ViT). When comparing across architectures configured to similar parameter complexity, ViTCA architectures yield superior performance across all benchmarks and for nearly every evaluation metric. We present an ablation study on various architectural configurations of ViTCA, an analysis of its effect on cell states, and an investigation on its inductive biases. Finally, we examine its learned representations via linear probes on its converged cell state hidden representations, yielding, on average, superior results when compared to our U-Net, ViT, and UNetCA baselines.