# Emergence and Function of Abstract Representations in Self-Supervised Transformers arXiv: https://arxiv.org/abs/2312.05361 ยท HTML: https://mlsanity.com/paper/2312.05361/ ## Key facts - Cited by 3 papers in the map (data through 14 Oct 2026). - Placed in World models for language agents, one of 11 directions in World models. - Builds on 39 of its 71 references that are in the map. - First posted on arXiv 8 Dec 2023. ## Placed in - World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/) ## Builds on (most cited first, 30 of 39) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020) - [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023) - [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021) - [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016) - [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017) - [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022) - [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022) - [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022) - [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022) - [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021) - [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022) - [Faith and Fate: Limits of Transformers on Compositionality](https://mlsanity.com/paper/2305.18654/) (2023) - [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020) - [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022) - [Causal Abstractions of Neural Networks](https://mlsanity.com/paper/2106.02997/) (2021) - [Emergent Linear Representations in World Models of Self-Supervised Sequence Models](https://mlsanity.com/paper/2309.00941/) (2023) - [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023) - [In-Context Learning Creates Task Vectors](https://mlsanity.com/paper/2310.15916/) (2023) - [Function Vectors in Large Language Models](https://mlsanity.com/paper/2310.15213/) (2023) - [Cross-Entropy Loss Functions: Theoretical Analysis and Applications](https://mlsanity.com/paper/2304.07288/) (2023) - [Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations](https://mlsanity.com/paper/2303.02536/) (2023) - [The Computational Limits of Deep Learning](https://mlsanity.com/paper/2007.05558/) (2020) - [Linear Representations of Sentiment in Large Language Models](https://mlsanity.com/paper/2310.15154/) (2023) - [The Hydra Effect: Emergent Self-repair in Language Model Computations](https://mlsanity.com/paper/2307.15771/) (2023) - [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022) - [How to represent part-whole hierarchies in a neural network](https://mlsanity.com/paper/2102.12627/) (2021) ## Built on by (most cited first, 3 of 3) - [Finding Visual Task Vectors](https://mlsanity.com/paper/2404.05729/) (2024) - [Representation Engineering for Large-Language Models: Survey and Research Challenges](https://mlsanity.com/paper/2502.17601/) (2025) - [Large Language Models and Emergence: A Complex Systems Perspective](https://mlsanity.com/paper/2506.11135/) (2025) ## Abstract (arXiv) Human intelligence relies in part on our brains' ability to create abstract mental models that succinctly capture the hidden blueprint of our reality. Such abstract world models notably allow us to rapidly navigate novel situations by generalizing prior knowledge, a trait deep learning systems have historically struggled to replicate. However, the recent shift from supervised to self-supervised objectives, combined with expressive transformer-based architectures, have yielded powerful foundation models that appear to learn versatile representations that can support a wide range of downstream tasks. This promising development raises the intriguing possibility of such models developing in silico abstract world models. We test this hypothesis by studying the inner workings of small-scale transformers trained to reconstruct partially masked visual scenes generated from a simple blueprint. We show that the network develops intermediate abstract representations, or abstractions, that encode all semantic features of the dataset. These abstractions manifest as low-dimensional manifolds where the embeddings of semantically related tokens transiently converge, thus allowing for the generalization of downstream computations. Using precise manipulation experiments, we demonstrate that abstractions are central to the network's decision-making process. Our research also suggests that these abstractions are compositionally structured, exhibiting features like contextual independence and part-whole relationships that mirror the compositional nature of the dataset. Finally, we introduce a Language-Enhanced Architecture (LEA) designed to encourage the network to articulate its computations. We find that LEA develops an abstraction-centric language that can be easily interpreted, allowing us to more readily access and steer the network's decision-making process.