mlsanityData to 7 Oct 2026

Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla

arXiv 2307.09458 ↗First posted 18 Jul 2023cs.LGcited by 145

Not on the map. Why?

Abstract (arXiv)

Circuit analysis is a promising technique for understanding the internal mechanisms of language models. However, existing analyses are done in small models far from the state of the art. To address this, we present a case study of circuit analysis in the 70B Chinchilla model, aiming to test the scalability of circuit analysis. In particular, we study multiple-choice question answering, and investigate Chinchilla's capability to identify the correct answer label given knowledge of the correct answer text. We find that the existing techniques of logit attribution, attention pattern visualization, and activation patching naturally scale to Chinchilla, allowing us to identify and categorize a small set of `output nodes' (attention heads and MLPs). We further study the `correct letter' category of attention heads aiming to understand the semantics of their features, with mixed results. For normal multiple-choice question answers, we significantly compress the query, key and value subspaces of the head without loss of performance when operating on the answer labels for multiple-choice questions, and we show that the query and key subspaces represent an `Nth item in an enumeration' feature to at least some extent. However, when we attempt to use this explanation to understand the heads' behaviour on a more general distribution including randomized answer labels, we find that it is only a partial explanation, suggesting there is more to learn about the operation of `correct letter' heads on multiple choice question answering.

Key facts. Cited by 145 papers in the map (data through 7 Oct 2026). Builds on 37 of its 54 references that are in the map. First posted on arXiv 18 Jul 2023.

Builds on

37 of 54 references.

References · 37

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  8. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  11. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  12. 2022Red Teaming Language Models with Language Modelscited by 983
25 more
  1. 2022In-context Learning and Induction Headscited by 930
  2. 2022Toy Models of Superpositioncited by 928
  3. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  4. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  5. 2019Are Sixteen Heads Really Better than One?cited by 803
  6. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  7. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  8. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  9. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  10. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  11. 2021Causal Abstractions of Neural Networkscited by 401
  12. 2018AI safety via debatecited by 384
  13. 2022Self-critiquing models for assisting human evaluatorscited by 372
  14. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  15. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  16. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 197
  17. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
  18. 2022Language model compression with weighted low-rank factorizationcited by 192
  19. 2021Alignment of Language Agentscited by 167
  20. 2023Model evaluation for extreme riskscited by 167
  21. 2022Analyzing Transformers in Embedding Spacecited by 131
  22. 2023Interpretability at Scale: Identifying Causal Mechanisms in Alpacacited by 102
  23. 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformationscited by 89
  24. 2021Low-Complexity Probing via Finding Subnetworkscited by 63
  25. 2022What Are You Token About? Dense Retrieval as Distributions Over the Vocabularycited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 67; 2025: 39; 2026: 24.

145 papers.

Sort:

By area:

  1. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  2. 2023Explainability for Large Language Models: A Surveycited by 331
  3. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  4. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
  5. 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 196
  6. 2024How to use and interpret activation patchingcited by 191
  7. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 186
  8. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 165
  9. 2024Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanismscited by 144
  10. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 140
  11. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  12. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
38 more
  1. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 111
  2. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 110
  3. 2023Circuit Component Reuse Across Tasks in Transformer Language Modelscited by 105
  4. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 100
  5. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  6. 2024Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Controlcited by 83
  7. 2024RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representationscited by 77
  8. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 76
  9. 2023Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patchingcited by 75
  10. 2024On the Role of Attention Heads in Large Language Model Safetycited by 70
  11. 2024Attention Heads of Large Language Models: A Surveycited by 65
  12. 2024LLM Circuit Analyses Are Consistent Across Training and Scalecited by 60
  13. 2024Finding Transformer Circuits with Edge Pruningcited by 58
  14. 2024Confidence Regulation Neurons in Language Modelscited by 57
  15. 2024LoFiT: Localized Fine-tuning on LLM Representationscited by 56
  16. 2024From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuningcited by 51
  17. 2025MIB: A Mechanistic Interpretability Benchmarkcited by 49
  18. 2024Identifying Semantic Induction Heads to Understand In-Context Learningcited by 46
  19. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  20. 2024Interpreting Attention Layer Outputs with Sparse Autoencoderscited by 43
  21. 2024Interpreting and Improving Large Language Models in Arithmetic Calculationcited by 43
  22. 2024Softmax is not Enough (for Sharp Size Generalisation)cited by 42
  23. 2025Weight-sparse transformers have interpretable circuitscited by 40
  24. 2024Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficientcited by 39
  25. 2023Neuron-Level Knowledge Attribution in Large Language Modelscited by 37
  26. 2025Language Models use Lookbacks to Track Beliefscited by 36
  27. 2024Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questionscited by 35
  28. 2024Interpreting Arithmetic Mechanism in Large Language Models through Comparative Neuron Analysiscited by 31
  29. 2024Interpreting Key Mechanisms of Factual Recall in Transformer-Based Language Modelscited by 30
  30. 2024Hypothesis Testing the Circuit Hypothesis in LLMscited by 29
  31. 2025Steering off Course: Reliability Challenges in Steering Language Modelscited by 29
  32. 2024Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPTcited by 27
  33. 2024The Same But Different: Structural Similarities and Differences in Multilingual Language Modelingcited by 27
  34. 2024Mechanistic?cited by 25
  35. 2023Interpretability Illusions in the Generalization of Simplified Modelscited by 24
  36. 2024InterpBench: Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniquescited by 24
  37. 2025Mechanistic Interpretability of Emotion Inference in Large Language Modelscited by 24
  38. 2023Identifying and Adapting Transformer-Components Responsible for Gender Bias in an English Language Modelcited by 22

50 most cited of 145 All 145 (JSON)