mlsanityData to 7 Oct 2026

Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms

arXiv 2403.17806 ↗First posted 26 Mar 2024cs.LG cs.CLcited by 144

Not on the map. Why?

Abstract (arXiv)

Many recent language model (LM) interpretability studies have adopted the circuits framework, which aims to find the minimal computational subgraph, or circuit, that explains LM behavior on a given task. Most studies determine which edges belong in a LM's circuit by performing causal interventions on each edge independently, but this scales poorly with model size. Edge attribution patching (EAP), gradient-based approximation to interventions, has emerged as a scalable but imperfect solution to this problem. In this paper, we introduce a new method - EAP with integrated gradients (EAP-IG) - that aims to better maintain a core property of circuits: faithfulness. A circuit is faithful if all model edges outside the circuit can be ablated without changing the model's performance on the task; faithfulness is what justifies studying circuits, rather than the full model. Our experiments demonstrate that circuits found using EAP are less faithful than those found using EAP-IG, even though both have high node overlap with circuits found previously using causal interventions. We conclude more generally that when using circuits to compare the mechanisms models use to solve tasks, faithfulness, not overlap, is what should be measured.

Key facts. Cited by 144 papers in the map (data through 7 Oct 2026). Builds on 21 of its 37 references that are in the map. First posted on arXiv 26 Mar 2024.

Builds on

21 of 37 references.

References · 21

  1. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  3. 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
  4. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  5. 2022In-context Learning and Induction Headscited by 930
  6. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  7. 2021Causal Abstractions of Neural Networkscited by 401
  8. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  9. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  10. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  11. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  12. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
9 more
  1. 2023Linear Representations of Sentiment in Large Language Modelscited by 157
  2. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  3. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 140
  4. 2023Circuit Component Reuse Across Tasks in Transformer Language Modelscited by 105
  5. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 100
  6. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 88
  7. 2021Refining Targeted Syntactic Evaluation of Language Modelscited by 41
  8. 2023Interpretability Illusions in the Generalization of Simplified Modelscited by 24
  9. 2023Identifying and Adapting Transformer-Components Responsible for Gender Bias in an English Language Modelcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 22; 2025: 53; 2026: 69.

144 papers.

Sort:

By area:

  1. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  2. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 186
  3. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  4. 2024Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristicscited by 94
  5. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  6. 2024Knowledge Circuits in Pretrained Transformerscited by 60
  7. 2024LLM Circuit Analyses Are Consistent Across Training and Scalecited by 60
  8. 2024Finding Transformer Circuits with Edge Pruningcited by 58
  9. 2025MIB: A Mechanistic Interpretability Benchmarkcited by 49
  10. 2025Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMscited by 35
  11. 2024Activation Scaling for Steering and Interpreting Language Modelscited by 33
  12. 2025Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable?cited by 32
38 more
  1. 2025Towards Understanding Subliminal Learning: When and How Hidden Biases Transfercited by 30
  2. 2024The Same But Different: Structural Similarities and Differences in Multilingual Language Modelingcited by 27
  3. 2025MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connectionscited by 27
  4. 2025Training Language Models to Explain Their Own Computationscited by 27
  5. 2024Mechanistic?cited by 25
  6. 2024The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysiscited by 24
  7. 2024Efficient Automated Circuit Discovery in Transformers using Contextual Decompositioncited by 23
  8. 2025How Do LLMs Acquire New Knowledge? A Knowledge Circuits Perspective on Continual Pre-Trainingcited by 22
  9. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 22
  10. 2025EAP-GP: Mitigating Saturation Effect in Gradient-based Automated Circuit Identificationcited by 21
  11. 2025Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit Analysiscited by 20
  12. 2026Language Model Circuits Are Sparse in the Neuron Basiscited by 18
  13. 2025Position-aware Automatic Circuit Discoverycited by 14
  14. 2025Does Time Have Its Place? Temporal Heads: Where Language Models Recall Time-specific Informationcited by 14
  15. 2025Mechanistic Interpretability as Statistical Estimation: A Variance Analysiscited by 14
  16. 2024Sparse Autoencoders Enable Scalable and Reliable Circuit Identification in Language Modelscited by 13
  17. 2024Sheaf Discovery with Joint Computation Graph Pruning and Flexible Granularitycited by 11
  18. 2025Mechanistic Unveiling of Transformer Circuits: Self-Influence as a Key to Model Reasoningcited by 11
  19. 2025Are formal and functional linguistic mechanisms dissociated in language models?cited by 11
  20. 2025The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentationcited by 11
  21. 2025The Validation Gap: A Mechanistic Analysis of How Language Models Compute Arithmetic but Fail to Validate Itcited by 10
  22. 2025Contextualize-then-Aggregate: Circuits for In-Context Learning in Gemma-2 2Bcited by 10
  23. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  24. 2026Formal Mechanistic Interpretability: Automated Circuit Discovery with Provable Guaranteescited by 10
  25. 2024Circuit Compositions: Exploring Modular Structures in Transformer-Based Language Modelscited by 9
  26. 2025When and How Does CLIP Enable Domain and Compositional Generalization?cited by 9
  27. 2026The Curse of Multiple Mediators: Hidden Interaction Effects in Activation Patchingcited by 9
  28. 2025Evolution of Concepts in Language Model Pre-Trainingcited by 8
  29. 2025Reinforcement Learning Fine-Tuning Enhances Activation Intensity and Diversity in the Internal Circuitry of LLMscited by 8
  30. 2025Thinking Sparks!: Emergent Attention Heads in Reasoning Models During Post Trainingcited by 8
  31. 2026All Circuits Lead to Rome: Rethinking Functional Anisotropy in Circuit and Sheaf Discovery for LLMscited by 8
  32. 2026What Drives Representation Steering? A Mechanistic Case Study on Steering Refusalcited by 7
  33. 2026LLMs Know They're Wrong and Agree Anyway: The Shared Sycophancy-Lying Circuitcited by 7
  34. 2025Circuit Stability Characterizes Language Model Generalizationcited by 6
  35. 2025Crosscoding Through Time: Tracking Emergence & Consolidation Of Linguistic Representations Throughout LLM Pretrainingcited by 6
  36. 2025Unboxing the Black Box: Mechanistic Interpretability for Algorithmic Understanding of Neural Networkscited by 6
  37. 2025Beyond Components: Singular Vector-Based Interpretability of Transformer Circuitscited by 6
  38. 2026CURE:Circuit-Aware Unlearning for LLM-based Recommendationcited by 6

50 most cited of 144 All 144 (JSON)