# HyperDAS: Towards Automating Mechanistic Interpretability with Hypernetworks

arXiv: https://arxiv.org/abs/2503.10894 · HTML: https://mlsanity.com/paper/2503.10894/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability.
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 13 Mar 2025.

## Placed in

- Mechanistic interpretability / [Circuits and mechanistic analysis](https://mlsanity.com/mechinterp/circuits-and-mechanistic-analysis/)

## Builds on (most cited first, 23 of 23)

- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models](https://mlsanity.com/paper/2401.06102/) (2024)
- [Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations](https://mlsanity.com/paper/2303.02536/) (2023)
- [Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability](https://mlsanity.com/paper/2301.04709/) (2023)
- [Compositional Explanations of Neurons](https://mlsanity.com/paper/2006.14032/) (2020)
- [Inspecting and Editing Knowledge Representations in Language Models](https://mlsanity.com/paper/2304.00740/) (2023)
- [Natural Language Descriptions of Deep Visual Features](https://mlsanity.com/paper/2201.11114/) (2022)
- [Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models](https://mlsanity.com/paper/2106.06087/) (2021)
- [Interpretability at Scale: Identifying Causal Mechanisms in Alpaca](https://mlsanity.com/paper/2305.08809/) (2023)
- [Information Flow Routes: Automatically Interpreting Language Models at Scale](https://mlsanity.com/paper/2403.00824/) (2024)
- [A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis](https://mlsanity.com/paper/2305.15054/) (2023)
- [RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations](https://mlsanity.com/paper/2402.17700/) (2024)
- [Identifying Functionally Important Features with End-to-End Sparse Dictionary Learning](https://mlsanity.com/paper/2405.12241/) (2024)
- [A Multimodal Automated Interpretability Agent](https://mlsanity.com/paper/2404.14394/) (2024)
- [Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representations](https://mlsanity.com/paper/2408.10920/) (2024)
- [Rigorously Assessing Natural Language Explanations of Neurons](https://mlsanity.com/paper/2309.10312/) (2023)
- [FIND: A Function Description Benchmark for Evaluating Interpretability Methods](https://mlsanity.com/paper/2309.03886/) (2023)
- [Automatic Discovery of Visual Circuits](https://mlsanity.com/paper/2404.14349/) (2024)

## Built on by (most cited first, 9 of 9)

- [The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?](https://mlsanity.com/paper/2507.08802/) (2025)
- [HyperSteer: Activation Steering at Scale with Hypernetworks](https://mlsanity.com/paper/2506.03292/) (2025)
- [Predictive Concept Decoders: Training Scalable End-to-End Interpretability Assistants](https://mlsanity.com/paper/2512.15712/) (2025)
- [From Directions to Regions: Decomposing Activations in Language Models via Local Geometry](https://mlsanity.com/paper/2602.02464/) (2026)
- [Internal Causal Mechanisms Robustly Predict Language Model Out-of-Distribution Behaviors](https://mlsanity.com/paper/2505.11770/) (2025)
- [Formal Mechanistic Interpretability: Automated Circuit Discovery with Provable Guarantees](https://mlsanity.com/paper/2602.16823/) (2026)
- [Mechanistic Indicators of Steering Effectiveness in Large Language Models](https://mlsanity.com/paper/2602.01716/) (2026)
- [PLOT: Progressive Localization via Optimal Transport in Neural Causal Abstraction](https://mlsanity.com/paper/2605.06979/) (2026)
- [Capacity, Responsiveness and Alignment: What Makes a Latent Structure Actionable](https://mlsanity.com/paper/2610.06897/) (2026)

## Abstract (arXiv)

Mechanistic interpretability has made great strides in identifying neural network features (e.g., directions in hidden activation space) that mediate concepts(e.g., the birth year of a person) and enable predictable manipulation. Distributed alignment search (DAS) leverages supervision from counterfactual data to learn concept features within hidden states, but DAS assumes we can afford to conduct a brute force search over potential feature locations. To address this, we present HyperDAS, a transformer-based hypernetwork architecture that (1) automatically locates the token-positions of the residual stream that a concept is realized in and (2) constructs features of those residual stream vectors for the concept. In experiments with Llama3-8B, HyperDAS achieves state-of-the-art performance on the RAVEL benchmark for disentangling concepts in hidden states. In addition, we review the design decisions we made to mitigate the concern that HyperDAS (like all powerful interpretabilty methods) might inject new information into the target model rather than faithfully interpreting it.