# Infusing Lattice Symmetry Priors in Attention Mechanisms for Sample-Efficient Abstract Geometric Reasoning

arXiv: https://arxiv.org/abs/2306.03175 · HTML: https://mlsanity.com/paper/2306.03175/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 30 of its 46 references that are in the map.
- First posted on arXiv 5 Jun 2023.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision](https://mlsanity.com/paper/1904.12584/) (2019)
- [On the Relationship between Self-Attention and Convolutional Layers](https://mlsanity.com/paper/1911.03584/) (2019)
- [Compositional Attention Networks for Machine Reasoning](https://mlsanity.com/paper/1803.03067/) (2018)
- [COTR: Correspondence Transformer for Matching Across Images](https://mlsanity.com/paper/2103.14167/) (2021)
- [DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning](https://mlsanity.com/paper/2006.08381/) (2020)
- [EquiBind: Geometric Deep Learning for Drug Binding Structure Prediction](https://mlsanity.com/paper/2202.05146/) (2022)
- [Inferring and Executing Programs for Visual Reasoning](https://mlsanity.com/paper/1705.03633/) (2017)
- [Learning by Abstraction: The Neural State Machine](https://mlsanity.com/paper/1907.03950/) (2019)
- [The Consciousness Prior](https://mlsanity.com/paper/1709.08568/) (2017)
- [Building machines that adapt and compute like brains](https://mlsanity.com/paper/1711.04203/) (2017)
- [SPECTRE: Spectral Conditioning Helps to Overcome the Expressivity Limits of One-shot Graph Generators](https://mlsanity.com/paper/2204.01613/) (2022)
- [CoMIR: Contrastive Multimodal Image Representation for Registration](https://mlsanity.com/paper/2006.06325/) (2020)
- [Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale](https://mlsanity.com/paper/2203.00633/) (2022)
- [Text-based RL Agents with Commonsense Knowledge: New Challenges, Environments and Baselines](https://mlsanity.com/paper/2010.03790/) (2020)
- [Neural Execution Engines: Learning to Execute Subroutines](https://mlsanity.com/paper/2006.08084/) (2020)
- [Communicating Natural Programs to Humans and Machines](https://mlsanity.com/paper/2106.07824/) (2021)
- [Convolution, attention and structure embedding](https://mlsanity.com/paper/1905.01289/) (2019)
- [Strong Generalization and Efficiency in Neural Programs](https://mlsanity.com/paper/2007.03629/) (2020)
- [SQALER: Scaling Question Answering by Decoupling Multi-Hop and Logical Reasoning](https://mlsanity.com/paper/2110.14266/) (2021)
- [Is Image-to-Image Translation the Panacea for Multimodal Image Registration? A Comparative Study](https://mlsanity.com/paper/2103.16262/) (2021)
- [Neural Abstract Reasoner](https://mlsanity.com/paper/2011.09860/) (2020)

## Built on by (most cited first, 4 of 4)

- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [The ARC of Progress towards AGI: A Living Survey of Abstraction and Reasoning](https://mlsanity.com/paper/2603.13372/) (2026)
- [System 2 Reasoning for Human-AI Alignment: Generality and Adaptivity via ARC-AGI](https://mlsanity.com/paper/2410.07866/) (2024)
- [Attention Function as an Intrinsic Inductive Bias: How Models' Behavior Diverges in Novel Contexts](https://mlsanity.com/paper/2609.39188/) (2026)

## Abstract (arXiv)

The Abstraction and Reasoning Corpus (ARC) (Chollet, 2019) and its most recent language-complete instantiation (LARC) has been postulated as an important step towards general AI. Yet, even state-of-the-art machine learning models struggle to achieve meaningful performance on these problems, falling behind non-learning based approaches. We argue that solving these tasks requires extreme generalization that can only be achieved by proper accounting for core knowledge priors. As a step towards this goal, we focus on geometry priors and introduce LatFormer, a model that incorporates lattice symmetry priors in attention masks. We show that, for any transformation of the hypercubic lattice, there exists a binary attention mask that implements that group action. Hence, our study motivates a modification to the standard attention mechanism, where attention weights are scaled using soft masks generated by a convolutional network. Experiments on synthetic geometric reasoning show that LatFormer requires 2 orders of magnitude fewer data than standard attention and transformers. Moreover, our results on ARC and LARC tasks that incorporate geometric priors provide preliminary evidence that these complex datasets do not lie out of the reach of deep learning models.