mlsanityData to 7 Oct 2026

Challenges in Mechanistically Interpreting Model Representations

arXiv 2402.03855 ↗First posted 6 Feb 2024cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Mechanistic interpretability (MI) aims to understand AI models by reverse-engineering the exact algorithms neural networks learn. Most works in MI so far have studied behaviors and capabilities that are trivial and token-aligned. However, most capabilities important for safety and trust are not that trivial, which advocates for the study of hidden representations inside these networks as the unit of analysis. We formalize representations for features and behaviors, highlight their importance and evaluation, and perform an exploratory study of dishonesty representations in `Mistral-7B-Instruct-v0.1'. We justify that studying representations is an important and under-studied field, and highlight several challenges that arise while attempting to do so through currently established methods in MI, showing their insufficiency and advocating work on new frameworks for the same.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 22 of its 38 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

22 of 38 references.

References · 22

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  3. 2023Mistral 7Bcited by 3,065
  4. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  5. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  6. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  7. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  8. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  9. 2022In-context Learning and Induction Headscited by 930
  10. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  11. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  12. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
10 more
  1. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  2. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  3. 2023Function Vectors in Large Language Modelscited by 292
  4. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  5. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 197
  6. 2023Linear Representations of Sentiment in Large Language Modelscited by 157
  7. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  8. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  9. 2023A Survey on Fairness in Large Language Modelscited by 91
  10. 2023Simple Mechanisms for Representing, Indexing and Manipulating Conceptscited by 1

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2025The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?cited by 34
  2. 2024Locking Down the Finetuned LLMs Safetycited by 22
  3. 2025Interpretability as Alignment: Making Internal Understanding a Design Principlecited by 3
  4. 2024Training Neural Networks for Modularity aids Interpretabilitycited by 1