# Make Mechanistic Interpretability Auditable: A Call to Develop Guidelines via Continuous Collaborative Reviewing arXiv: https://arxiv.org/abs/2606.00033 ยท HTML: https://mlsanity.com/paper/2606.00033/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability. - Builds on 31 of its 80 references that are in the map. - First posted on arXiv 24 Apr 2026. ## Placed in - Mechanistic interpretability / [Circuits and mechanistic analysis](https://mlsanity.com/mechinterp/circuits-and-mechanistic-analysis/) ## Builds on (most cited first, 30 of 31) - [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022) - [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022) - [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018) - [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021) - [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023) - [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024) - [Neural Probabilistic Logic Programming in DeepProbLog](https://mlsanity.com/paper/1907.08194/) (2019) - [Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning](https://mlsanity.com/paper/2305.12295/) (2023) - [Towards Best Practices of Activation Patching in Language Models: Metrics and Methods](https://mlsanity.com/paper/2309.16042/) (2023) - [LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers](https://mlsanity.com/paper/2310.15164/) (2023) - [End-to-End Differentiable Proving](https://mlsanity.com/paper/1705.11040/) (2017) - [A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders](https://mlsanity.com/paper/2409.14507/) (2024) - [The Hydra Effect: Emergent Self-repair in Language Model Computations](https://mlsanity.com/paper/2307.15771/) (2023) - [A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations](https://mlsanity.com/paper/2302.03025/) (2023) - [Analyzing the Generalization and Reliability of Steering Vectors](https://mlsanity.com/paper/2407.12404/) (2024) - [Tracr: Compiled Transformers as a Laboratory for Interpretability](https://mlsanity.com/paper/2301.05062/) (2023) - [LLM Circuit Analyses Are Consistent Across Training and Scale](https://mlsanity.com/paper/2407.10827/) (2024) - [The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?](https://mlsanity.com/paper/2507.08802/) (2025) - [Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers](https://mlsanity.com/paper/2406.14325/) (2024) - [Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable?](https://mlsanity.com/paper/2502.20914/) (2025) - [Real-Time Detection of Hallucinated Entities in Long-Form Generation](https://mlsanity.com/paper/2509.03531/) (2025) - [Hypothesis Testing the Circuit Hypothesis in LLMs](https://mlsanity.com/paper/2410.13032/) (2024) - [Interpretability Illusions in the Generalization of Simplified Models](https://mlsanity.com/paper/2312.03656/) (2023) - [InterpBench: Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniques](https://mlsanity.com/paper/2407.14494/) (2024) - [Position: Mechanistic Interpretability Should Prioritize Feature Consistency in SAEs](https://mlsanity.com/paper/2505.20254/) (2025) - [The Dead Salmons of AI Interpretability](https://mlsanity.com/paper/2512.18792/) (2025) - [Towards a unified and verified understanding of group-operation networks](https://mlsanity.com/paper/2410.07476/) (2024) - [Position: Embracing Negative Results in Machine Learning](https://mlsanity.com/paper/2406.03980/) (2024) - [ReproducedPapers.org: Openly teaching and structuring machine learning reproducibility](https://mlsanity.com/paper/2012.01172/) (2020) - [DILA: Dictionary Label Attention for Mechanistic Interpretability in High-dimensional Multi-label Medical Coding Prediction](https://mlsanity.com/paper/2409.10504/) (2024) ## Built on by (most cited first, 2 of 2) - [Explanation Multiplicity: Circuit-Level Interpretability Evidence Does Not Survive Defensible Analytic Variation](https://mlsanity.com/paper/2608.13754/) (2026) - [Necessary, Decodable and Reversible, Yet Not Transferable: A Stress Test for Attention-Head Role Claims](https://mlsanity.com/paper/2606.08292/) (2026) ## Abstract (arXiv) While mechanistic interpretability (MI) has produced important insights into neural network internals, the field has yet to establish a standardized system to audit experiments. As such, many of its findings remain underutilized in safety-critical applications such as medical AI and autonomous systems, as stakeholders cannot certify their validity. Recent work demonstrates this concretely: two papers found conflicting conclusions for the same behavior, and a third study revealed that both were partially correct but incomparable due to methodological inconsistencies. Without standardized auditing, such ambiguities hinder adoption in high-stakes contexts requiring strong correctness guarantees. We call for the MI community to work towards developing a novel reviewing system that complements peer review via: (1) Continuous reviewing supported by a Collaborative Reviewing Platform where meta-science results and discussions (such as critiques, negative results, post-hoc extensions, reproductions, replications, and partial results) that fit outside of papers are organized and discussed, allowing for comments and revisions to be made at any time (2) Generalizing good practices found on this platform into expert-verified guidelines and protocols to improve auditing efficiency, and (3) Source-based auditing systems that track arguments which claims depend on. This position paper encourages constructive debate over the necessity, design and implementation of such a framework, providing early concrete examples to help catalyze these dialogues. Overall, we propose that auditing MI itself is essential for its application in AI safety, industry, and governance.