# Differentiable Faithfulness Alignment for Cross-Model Circuit Transfer

arXiv: https://arxiv.org/abs/2604.24302 · HTML: https://mlsanity.com/paper/2604.24302/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability.
- Builds on 19 of its 22 references that are in the map.
- First posted on arXiv 27 Apr 2026.

## Placed in

- Mechanistic interpretability / [Circuits and mechanistic analysis](https://mlsanity.com/mechinterp/circuits-and-mechanistic-analysis/)

## Builds on (most cited first, 19 of 19)

- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations](https://mlsanity.com/paper/2303.02536/) (2023)
- [Attribution Patching Outperforms Automated Circuit Discovery](https://mlsanity.com/paper/2310.10348/) (2023)
- [Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla](https://mlsanity.com/paper/2307.09458/) (2023)
- [Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking](https://mlsanity.com/paper/2402.14811/) (2024)
- [Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms](https://mlsanity.com/paper/2403.17806/) (2024)
- [Circuit Component Reuse Across Tasks in Transformer Language Models](https://mlsanity.com/paper/2310.08744/) (2023)
- [LLM Circuit Analyses Are Consistent Across Training and Scale](https://mlsanity.com/paper/2407.10827/) (2024)
- [MIB: A Mechanistic Interpretability Benchmark](https://mlsanity.com/paper/2504.13151/) (2025)
- [Similarity Analysis of Contextual Word Representation Models](https://mlsanity.com/paper/2005.01172/) (2020)
- [Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs](https://mlsanity.com/paper/2506.09047/) (2025)
- [Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures](https://mlsanity.com/paper/2410.06672/) (2024)
- [Towards Interpretable Sequence Continuation: Analyzing Shared Circuits in Large Language Models](https://mlsanity.com/paper/2311.04131/) (2023)
- [Layer by Layer: Uncovering Where Multi-Task Learning Happens in Instruction-Tuned Large Language Models](https://mlsanity.com/paper/2410.20008/) (2024)
- [Circuit Compositions: Exploring Modular Structures in Transformer-Based Language Models](https://mlsanity.com/paper/2410.01434/) (2024)
- [A Joint Matrix Factorization Analysis of Multilingual Representations](https://mlsanity.com/paper/2310.15513/) (2023)
- [Toward a Theory of Generalizability in LLM Mechanistic Interpretability Research](https://mlsanity.com/paper/2509.22831/) (2025)

## Built on by (most cited first, 3 of 3)

- [Debiasing Without Protected Attributes: Latent Concept Erasure from Textual Profiles](https://mlsanity.com/paper/2606.12088/) (2026)
- [Can Graph Learning Learn Circuits?](https://mlsanity.com/paper/2608.08536/) (2026)
- [Shared circuits predict whether LLMs generalize across formats in arithmetic reasoning](https://mlsanity.com/paper/2609.04463/) (2026)

## Abstract (arXiv)

Mechanistic interpretability has made it possible to localize circuits underlying specific behaviors in language models, but existing methods are expensive, model-specific, and difficult to scale to larger architectures. We introduce Differentiable Faithfulness Alignment (DFA), a framework that transfers circuit information from a smaller source model to a larger target model through a learned differentiable alignment. DFA projects source-model node importance scores into the target model and trains this mapping with a soft faithfulness objective, avoiding full circuit discovery on the target model. We evaluate DFA on Llama-3 and Qwen-2.5 across six tasks spanning factual retrieval, multiple-choice reasoning, and arithmetic. The strongest results occur on Llama-3 1Brightarrow3B, where aligned circuits are often competitive with direct node attribution and zero-shot transfer remains effective. Recovery weakens for larger source--target gaps and is substantially lower on Qwen-2.5, suggesting that transfer becomes harder as architectural and scaling differences increase. Overall, DFA consistently outperforms simple baselines and, in some settings, recovers target-model circuits with faithfulness comparable to or stronger than direct attribution. These results suggest that smaller models can provide useful mechanistic priors for larger ones, while highlighting both the promise and the limits of node-level cross-model circuit alignment.footnoteCode is available at https://github.com/jasonshaoshun/dfa-circuits.