# Causal Masking on Spatial Data: An Information-Theoretic Case for Learning Spatial Datasets with Unimodal Language Models

arXiv: https://arxiv.org/abs/2510.27009 · HTML: https://mlsanity.com/paper/2510.27009/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 21 references that are in the map.
- First posted on arXiv 30 Oct 2025.

## Builds on (most cited first, 15 of 15)

- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Transformers represent belief state geometry in their residual stream](https://mlsanity.com/paper/2405.15943/) (2024)
- [Chess as a Testbed for Language Model State Tracking](https://mlsanity.com/paper/2102.13249/) (2021)
- [Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models](https://mlsanity.com/paper/2403.15498/) (2024)
- [Amortized Planning with Large-Scale Transformers: A Case Study on Chess](https://mlsanity.com/paper/2402.04494/) (2024)
- [The Chess Transformer: Mastering Play using Generative Language Models](https://mlsanity.com/paper/2008.04057/) (2020)
- [Mastering Board Games by External and Internal Planning with Language Models](https://mlsanity.com/paper/2412.12119/) (2024)
- [Linear Latent World Models in Simple Transformers: A Case Study on Othello-GPT](https://mlsanity.com/paper/2310.07582/) (2023)
- [Structured World Representations in Maze-Solving Transformers](https://mlsanity.com/paper/2312.02566/) (2023)
- [Complete Chess Games Enable LLM Become A Chess Master](https://mlsanity.com/paper/2501.17186/) (2025)
- [Transformers Use Causal World Models in Maze-Solving Tasks](https://mlsanity.com/paper/2412.11867/) (2024)
- [Implicit Search via Discrete Diffusion: A Study on Chess](https://mlsanity.com/paper/2502.19805/) (2025)
- [Rethinking Causal Mask Attention for Vision-Language Inference](https://mlsanity.com/paper/2505.18605/) (2025)
- [Sample- and Parameter-Efficient Auto-Regressive Image Models](https://mlsanity.com/paper/2411.15648/) (2024)
- [Learning to Plan via Supervised Contrastive Learning and Strategic Interpolation: A Chess Case Study](https://mlsanity.com/paper/2506.04892/) (2025)

## Abstract (arXiv)

Language models are traditionally designed around causal masking. In domains with spatial or relational structure, causal masking is often viewed as inappropriate, and sequential linearizations are instead used. Yet the question of whether it is viable to accept the information loss introduced by causal masking on nonsequential data has received little direct study, in part because few domains offer both spatial and sequential representations of the same dataset. In this work, we investigate this issue in the domain of chess, which naturally supports both representations. We train language models with bidirectional and causal self-attention mechanisms on both spatial (board-based) and sequential (move-based) data. Our results show that models trained on spatial board states - textiteven with causal masking - consistently achieve stronger playing strength than models trained on sequential data. While our experiments are conducted on chess, our results are methodological and may have broader implications: applying causal masking to spatial data is a viable procedure for training unimodal LLMs on spatial data, and in some domains is even preferable to sequentialization.