mlsanityData to 14 Oct 2026

Transformer Circuit Faithfulness Metrics are not Robust

arXiv 2407.08734 ↗First posted 11 Jul 2024cs.LG cs.AI cs.CLcited by 22

On the map: Mechanistic interpretability → Circuits and mechanistic analysis

Abstract (arXiv)

Mechanistic interpretability work attempts to reverse engineer the learned algorithms present inside neural networks. One focus of this work has been to discover 'circuits' -- subgraphs of the full model that explain behaviour on specific tasks. But how do we measure the performance of such circuits? Prior work has attempted to measure circuit 'faithfulness' -- the degree to which the circuit replicates the performance of the full model. In this work, we survey many considerations for designing experiments that measure circuit faithfulness by ablating portions of the model's computation. Concerningly, we find existing methods are highly sensitive to seemingly insignificant changes in the ablation methodology. We conclude that existing circuit faithfulness scores reflect both the methodological choices of researchers as well as the actual components of the circuit - the task a circuit is required to perform depends on the ablation used to test it. The ultimate goal of mechanistic interpretability work is to understand neural networks, so we emphasize the need for more clarity in the precise claims being made about circuits. We open source a library at https://github.com/UFO-101/auto-circuit that includes highly efficient implementations of a wide range of ablation methodologies and circuit discovery algorithms.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability. Builds on 22 of its 38 references that are in the map. First posted on arXiv 11 Jul 2024.

Builds on

22 of 38 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching28 Nov
  2. 2023Towards Automated Circuit Discovery for Mechanistic Interpretability28 Apr
  3. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations6 Feb
  4. 2023Progress measures for grokking via mechanistic interpretability12 Jan
  5. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small1 Nov

Other references · 17

  1. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  2. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  3. 2018A Survey Of Methods For Explaining Black Box Modelscited by 973
  4. 2022In-context Learning and Induction Headscited by 964
  5. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 599
  6. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  7. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 431
  8. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 359
  9. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 209
  10. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 202
  11. 2023Localizing Model Behavior with Path Patchingcited by 195
  12. 2021Thinking Like Transformerscited by 184
5 more
  1. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 111
  2. 2024Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Modelscited by 105
  3. 2023Tracr: Compiled Transformers as a Laboratory for Interpretabilitycited by 104
  4. 2023Circuit Breaking: Removing Model Behaviors with Targeted Ablationcited by 27
  5. 2019Analyzing and Interpreting Neural Networks for NLP: A Report on the First BlackboxNLP Workshopcited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 15.

22 papers.

Sort:

By area:

  1. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  2. 2025EAP-GP: Mitigating Saturation Effect in Gradient-based Automated Circuit Identificationcited by 21
  3. 2025Mechanistic Unveiling of Transformer Circuits: Self-Influence as a Key to Model Reasoningcited by 11
  4. 2025Are formal and functional linguistic mechanisms dissociated in language models?cited by 11
  5. 2024Towards a unified and verified understanding of group-operation networkscited by 10
  6. 2026Conditional Co-Ablation: Recovering Self-Repair Backups in Transformer Circuitscited by 9
  7. 2025Layerwise Recall and the Geometry of Interwoven Knowledge in LLMscited by 6
  8. 2025Scaling sparse feature circuit finding for in-context learningcited by 5
  9. 2026Certified Circuits: Stability Guarantees for Mechanistic Circuitscited by 5
  10. 2026Certified Interventional Fidelity: Anytime-Valid, Adaptive Evaluation of Causal Claims in Mechanistic Interpretabilitycited by 4
  11. 2025PAHQ: Accelerating Automated Circuit Discovery through Mixed-Precision Inference Optimizationcited by 3
  12. 2026Explanation Multiplicity: Circuit-Level Interpretability Evidence Does Not Survive Defensible Analytic Variationcited by 2
10 more
  1. 2026Sparse Weight Decomposition for Efficient Circuit Extractioncited by 1
  2. 2026Explaining the Explainer: Understanding the Inner Workings of Transformer-based Symbolic Regression Modelscited by 0
  3. 2026CircuitKIT : Circuit Discovery, Evaluation, and Application Toolkit for Mechanistic Interpretabilitycited by 0
  4. 2026Circuit Condensation: Post-Training that Concentrates a Behavior's Causal Circuitcited by 0
  5. 2026What Would Falsify It? A Variable Specific Evidence Standard for Mechanistic Claims About Self Explanationcited by 0
  6. 2026Are We Recovering Mechanisms? Objective-Level Recovery Gaps in Mechanistic Interpretabilitycited by 0
  7. 2026Every Ablation Is a Dose: Counterweights and the Semblance of Self-Repaircited by 0
  8. 2026Does the Model Use the Feature? Separating Steering from Mechanism in LLMscited by 0
  9. 2026FlyHack: Stress-Testing Mechanistic Claims in a Connectome-Constrained Visual Modelcited by 0
  10. 2026What Do Visual-Head Discovery Scores Actually Find? Attention Source, Necessity, and Random-Control Pitfalls in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Formal Mechanistic Interpretability: Automated Circuit Discovery with Provable Guarantees
  2. 2026Rethinking Circuit Evaluation: Do Circuits Explain Model Errors?
  3. 2026Seeing Through Circuits: Faithful Mechanistic Interpretability for Vision Transformers
  4. 2025Measuring Uncertainty in Transformer Circuits with Effective Information Consistency
  5. 2026Mechanistic Interpretability for Neural Networks: Circuits, Sparse Features and Symbolic Reasoning
  6. 2024Efficient Automated Circuit Discovery in Transformers using Contextual Decomposition
  7. 2024InterpBench: Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniques
  8. 2025Circuit Insights: Towards Interpretability Beyond Activations