What Is the Minimum Architecture for Prolepsis? Early Irrevocable Commitment Across Tasks in Small Transformers
Not on the map. Why?
Abstract (arXiv)
When do transformers commit to a decision, and what prevents them from correcting it? We introduce prolepsis: a transformer commits early, task-specific attention heads sustain the commitment, and no layer corrects it. Replicating Lindsey et al.'s (2025) planning-site finding on open models (Gemma 2 2B, Llama 3.2 1B), we ask five questions. (Q1) Planning is invisible to six residual-stream methods; among those tested, only CLT-based steering succeeds. (Q2) The single-site spike replicates in shape, at the final prompt token (Anthropic's site is the newline; see the Note added). (Q3) Specific attention heads route the decision to the output, filling a gap flagged as invisible to attribution graphs. (Q4) The evidence is consistent with search within at most 16 layers and commitment beyond, a two-model hypothesis. (Q5) Factual recall shows the same motif at a different network depth, with zero overlap between recurring planning heads and the factual top-10. Prolepsis recurs across tasks in the decoder-only models tested: the template is shared, the routing substrates differ. All experiments run on a single consumer GPU (16 GB VRAM).
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 17 of its 30 references that are in the map. First posted on arXiv 16 Apr 2026.
Builds on
17 of 30 references.
References · 17
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
- 2022In-context Learning and Induction Headscited by 930
- 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
- 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
- 2021Knowledge Neurons in Pretrained Transformerscited by 590
- 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
- 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
- 2021Causal Abstractions of Neural Networkscited by 401
- 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125
- 2024Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrappingcited by 89
- 2025Training Language Models to Explain Their Own Computationscited by 27
5 more
- 2024Planning in a recurrent neural network that plays Sokobancited by 17
- 2025Do Large Language Models Have a Planning Theory of Mind? Evidence from MindGames: a Multi-Step Persuasion Taskcited by 9
- 2026What's the plan? Metrics for implicit planning in LLMs and their application to rhyme generation and question answeringcited by 6
- 2026Latent Planning Emerges with Scalecited by 5
- 2025Path Channels and Plan Extension Kernels: a Mechanistic Description of Planning in a Sokoban RNNcited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.