# On the Intrinsic Limits of Transformer Image Embeddings in Non-Solvable Spatial Reasoning

arXiv: https://arxiv.org/abs/2601.03048 · HTML: https://mlsanity.com/paper/2601.03048/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 0 of its 38 references that are in the map.
- First posted on arXiv 6 Jan 2026.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Abstract (arXiv)

Vision Transformers (ViTs) excel in semantic recognition but exhibit systematic failures in spatial reasoning tasks such as mental rotation. While often attributed to data scale, this work argues that the limitation arises from the intrinsic circuit complexity of the architecture. By formalizing spatial understanding as learning a Group Homomorphism Problem -- where latent embeddings preserve the algebraic structure of physical transformations acting on images -- we identify a fundamental computational bottleneck. Specifically, for non-solvable groups (e.g., SO(3)), maintaining such structure-preserving embeddings is lowerbounded by the Word Problem, which is mathsfNC1-complete. In contrast, constant-depth ViTs with polynomial precision are strictly bounded by the complexity class mathsfTC0. Under the standard conjecture mathsfTC0 subsetneq mathsfNC1, a complexity boundary emerges: constant-depth architectures lack the logical depth required to capture non-solvable spatial structures in a single forward pass. To empirically validate this theoretical gap, we propose the Latent Space Algebra (LSA) benchmark, which reveals a significant degradation in ViT representations as the compositional depth of non-solvable tasks increases.