Circuit Fingerprints: How Answer Tokens Encode Their Geometrical Path
Not on the map. Why?
Abstract (arXiv)
Circuit discovery and activation steering in transformers have developed as separate research threads, yet both operate on the same representational space. Are they two views of the same underlying structure? We show they follow a single geometric principle: answer tokens, processed in isolation, encode the directions that would produce them. This Circuit Fingerprint hypothesis enables circuit discovery without gradients or causal intervention -- recovering comparable structure to gradient-based methods through geometric alignment alone. We validate this on standard benchmarks (IOI, SVA, MCQA) across four model families, achieving circuit discovery performance comparable to gradient-based methods. The same directions that identify circuit components also enable controlled steering -- achieving 69.8% emotion classification accuracy versus 53.1% for instruction prompting while preserving factual accuracy. Beyond method development, this read-write duality reveals that transformer circuits are fundamentally geometric structures: interpretability and controllability are two facets of the same object.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 24 of its 31 references that are in the map. First posted on arXiv 10 Feb 2026.
Builds on
24 of 31 references.
References · 24
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2024Qwen2.5 Technical Reportcited by 4,713
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
- 2016Understanding intermediate layers using linear classifier probescited by 1,279
- 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
- 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
- 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
- 2022Toy Models of Superpositioncited by 928
- 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
- 2023Steering Language Models With Activation Engineeringcited by 824
12 more
- 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
- 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
- 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
- 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
- 2023Language Models Represent Space and Timecited by 306
- 2023Function Vectors in Large Language Modelscited by 292
- 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 196
- 2024Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanismscited by 144
- 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 100
- 2024Finding Transformer Circuits with Edge Pruningcited by 58
- 2024Style Vectors for Steering Generative Large Language Modelcited by 56
- 2025MIB: A Mechanistic Interpretability Benchmarkcited by 49
Built on by
Citing papers per year: 2026: 1.
1 paper.