# Measuring Affinity between Attention-Head Weight Subspaces via the Projection Kernel

arXiv: https://arxiv.org/abs/2601.10266 · HTML: https://mlsanity.com/paper/2601.10266/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 64 references that are in the map.
- First posted on arXiv 15 Jan 2026.

## Builds on (most cited first, 30 of 32)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [Is Attention Interpretable?](https://mlsanity.com/paper/1906.03731/) (2019)
- [All-but-the-Top: Simple and Effective Postprocessing for Word Representations](https://mlsanity.com/paper/1702.01417/) (2017)
- [Learning to Deceive with Attention-Based Explanations](https://mlsanity.com/paper/1909.07913/) (2019)
- [Universal Neurons in GPT2 Language Models](https://mlsanity.com/paper/2401.12181/) (2024)
- [Information Flow Routes: Automatically Interpreting Language Models at Scale](https://mlsanity.com/paper/2403.00824/) (2024)
- [What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation](https://mlsanity.com/paper/2404.07129/) (2024)
- [Successor Heads: Recurring, Interpretable Attention Heads In The Wild](https://mlsanity.com/paper/2312.09230/) (2023)
- [Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin](https://mlsanity.com/paper/2510.06477/) (2025)
- [Talking Heads: Understanding Inter-layer Communication in Transformer Language Models](https://mlsanity.com/paper/2406.09519/) (2024)
- [Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures](https://mlsanity.com/paper/2410.06672/) (2024)
- [The Same But Different: Structural Similarities and Differences in Multilingual Language Modeling](https://mlsanity.com/paper/2410.09223/) (2024)
- [Robust subspace clustering](https://mlsanity.com/paper/1301.2603/) (2013)
- [Efficient Automated Circuit Discovery in Transformers using Contextual Decomposition](https://mlsanity.com/paper/2407.00886/) (2024)
- [From Balustrades to Pierre Vinken: Looking for Syntax in Transformer Self-Attentions](https://mlsanity.com/paper/1906.01958/) (2019)
- [You can remove GPT2's LayerNorm by fine-tuning](https://mlsanity.com/paper/2409.13710/) (2024)
- [Abrupt Learning in Transformers: A Case Study on Matrix Completion](https://mlsanity.com/paper/2410.22244/) (2024)
- [Axis Tour: Word Tour Determines the Order of Axes in ICA-transformed Embeddings](https://mlsanity.com/paper/2401.06112/) (2024)
- [Revisiting Cosine Similarity via Normalized ICA-transformed Embeddings](https://mlsanity.com/paper/2406.10984/) (2024)
- [Weight-based Analysis of Detokenization in Language Models: Understanding the First Stage of Inference Without Inference](https://mlsanity.com/paper/2501.15754/) (2025)

## Built on by (most cited first, 5 of 5)

- [Conditional Co-Ablation: Recovering Self-Repair Backups in Transformer Circuits](https://mlsanity.com/paper/2607.01940/) (2026)
- [Flag Varieties: A Geometric Framework for Deep Network Alignment](https://mlsanity.com/paper/2605.09861/) (2026)
- [The Communication Map of a Transformer](https://mlsanity.com/paper/2608.22007/) (2026)
- [Scaled Idempotence in Transformer Attention: Paired OV Geometry and Shared-Value Algebras](https://mlsanity.com/paper/2609.01129/) (2026)
- [The Linear Representation Hypothesis Needs a Group Action](https://mlsanity.com/paper/2609.27158/) (2026)

## Abstract (arXiv)

Understanding relationships between attention heads is essential for interpreting the internal structure of Transformers, yet existing metrics do not capture this structure well. We focus on the subspaces spanned by attention-head weight matrices and quantify head-to-head relationships using the Projection Kernel (PK), a principal-angle-based measure of subspace similarity. Experiments show that PK reproduces known head-to-head interactions on the IOI task more clearly than prior metrics such as the Composition Score. We further introduce a framework to quantify the informativeness of PK distributions by comparing them with a reference distribution derived from random orthogonal subspaces. As an application, we analyze a directed graph constructed from PK and show that, in GPT2-small, L4H7 acts as a hub by functioning as an identity head.