# Mechanistic?

arXiv: https://arxiv.org/abs/2410.09087 · HTML: https://mlsanity.com/paper/2410.09087/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability.
- Builds on 120 of its 203 references that are in the map.
- First posted on arXiv 7 Oct 2024.

## Placed in

- Mechanistic interpretability / [Circuits and mechanistic analysis](https://mlsanity.com/mechinterp/circuits-and-mechanistic-analysis/)

## Builds on (most cited first, 30 of 120)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)

## Built on by (most cited first, 25 of 25)

- [MIB: A Mechanistic Interpretability Benchmark](https://mlsanity.com/paper/2504.13151/) (2025)
- [Language Models use Lookbacks to Track Beliefs](https://mlsanity.com/paper/2505.14685/) (2025)
- [There Will Be a Scientific Theory of Deep Learning](https://mlsanity.com/paper/2604.21691/) (2026)
- [On Linear Representations and Pretraining Data Frequency in Language Models](https://mlsanity.com/paper/2504.12459/) (2025)
- [How Do Transformers Learn Variable Binding in Symbolic Programs?](https://mlsanity.com/paper/2505.20896/) (2025)
- [The Dead Salmons of AI Interpretability](https://mlsanity.com/paper/2512.18792/) (2025)
- [How Causal Abstraction Underpins Computational Explanation](https://mlsanity.com/paper/2508.11214/) (2025)
- [Adversarial Attacks Leverage Interference Between Features in Superposition](https://mlsanity.com/paper/2510.11709/) (2025)
- [Mechanistic Interpretability Needs Philosophy](https://mlsanity.com/paper/2506.18852/) (2025)
- [Unboxing the Black Box: Mechanistic Interpretability for Algorithmic Understanding of Neural Networks](https://mlsanity.com/paper/2511.19265/) (2025)
- [Interpretable Debiasing of Vision-Language Models for Social Fairness](https://mlsanity.com/paper/2602.24014/) (2026)
- [Aligned Probing: Relating Toxic Behavior and Model Internals](https://mlsanity.com/paper/2503.13390/) (2025)
- [VITAL: More Understandable Feature Visualization through Distribution Alignment and Relevant Information Flow](https://mlsanity.com/paper/2503.22399/) (2025)
- [ExPLAIND: Unifying Model, Data, and Training Attribution to Study Model Behavior](https://mlsanity.com/paper/2505.20076/) (2025)
- [Inner Loop Inference for Pretrained Transformers: Unlocking Latent Capabilities Without Training](https://mlsanity.com/paper/2602.14759/) (2026)
- [One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models](https://mlsanity.com/paper/2603.03291/) (2026)
- [How Open Must Language Models be to Enable Reliable Scientific Inference?](https://mlsanity.com/paper/2603.26539/) (2026)
- [Evaluating the Interpretability of Sparse Autoencoders with Concept Annotations](https://mlsanity.com/paper/2606.24716/) (2026)
- [Position: Explainability Research Must Prioritize Foundations over Ad-hoc Methods](https://mlsanity.com/paper/2607.14123/) (2026)
- [Constructive Circuit Amplification: Improving Math Reasoning in LLMs via Targeted Sub-Network Updates](https://mlsanity.com/paper/2512.16914/) (2025)
- [Residual Connections and the Causal Shift: Uncovering a Structural Misalignment in Transformers](https://mlsanity.com/paper/2602.14760/) (2026)
- [In-Context Learning in Speech Language Models: Analyzing the Role of Acoustic Features, Linguistic Structure, and Induction Heads](https://mlsanity.com/paper/2604.06356/) (2026)
- [A Hybrid Modeling Framework for Crop Prediction Tasks via Dynamic Parameter Calibration and Multi-Task Learning](https://mlsanity.com/paper/2603.15411/) (2026)
- [MI-MIDI: Mechanistic Interpretability of Text-to-MIDI Generation Models via Probing, Lenses and Steering](https://mlsanity.com/paper/2608.06638/) (2026)
- [Who Is Left of Whom? Tracing Spatial Evidence and Role Binding in Relative-Position Reasoning](https://mlsanity.com/paper/2609.35486/) (2026)

## Abstract (arXiv)

The rise of the term "mechanistic interpretability" has accompanied increasing interest in understanding neural models -- particularly language models. However, this jargon has also led to a fair amount of confusion. So, what does it mean to be "mechanistic"? We describe four uses of the term in interpretability research. The most narrow technical definition requires a claim of causality, while a broader technical definition allows for any exploration of a model's internals. However, the term also has a narrow cultural definition describing a cultural movement. To understand this semantic drift, we present a history of the NLP interpretability community and the formation of the separate, parallel "mechanistic" interpretability community. Finally, we discuss the broad cultural definition -- encompassing the entire field of interpretability -- and why the traditional NLP interpretability community has come to embrace it. We argue that the polysemy of "mechanistic" is the product of a critical divide within the interpretability community.