# Where Computation Lives Inside TabPFN: Causal Localisation of Attention Head Function

arXiv: https://arxiv.org/abs/2606.12917 · HTML: https://mlsanity.com/paper/2606.12917/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability.
- Builds on 10 of its 16 references that are in the map.
- First posted on arXiv 11 Jun 2026.

## Placed in

- Mechanistic interpretability / [Circuits and mechanistic analysis](https://mlsanity.com/mechinterp/circuits-and-mechanistic-analysis/)

## Builds on (most cited first, 10 of 10)

- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [How to use and interpret activation patching](https://mlsanity.com/paper/2404.15255/) (2024)
- [Why do tree-based models still outperform deep learning on tabular data?](https://mlsanity.com/paper/2207.08815/) (2022)
- [TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models](https://mlsanity.com/paper/2511.08667/) (2025)
- [Which Attention Heads Matter for In-Context Learning?](https://mlsanity.com/paper/2502.14010/) (2025)
- [A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities](https://mlsanity.com/paper/2502.17361/) (2025)
- [Exploring Representations and Interventions in Time Series Foundation Models](https://mlsanity.com/paper/2409.12915/) (2024)
- [In Search of Grandmother Cells: Tracing Interpretable Neurons in Tabular Representations](https://mlsanity.com/paper/2601.03657/) (2026)

## Built on by (most cited first, 4 of 4)

- [Gathered, Not Admitted: How Attention Brings a Latent Variable into Verbalizable Form](https://mlsanity.com/paper/2608.15022/) (2026)
- [Architecture Alignment With Sparse Priors in Tabular Foundation Models](https://mlsanity.com/paper/2609.36883/) (2026)
- [The Standardization Trap: Certifying Joint Label Processing in Tabular Foundation Models](https://mlsanity.com/paper/2610.08314/) (2026)
- [When Repeated Measurements Hurt Tabular Foundation Models](https://mlsanity.com/paper/2610.16801/) (2026)

## Abstract (arXiv)

We present the first causal mechanistic analysis of a tabular foundation model, investigating how TabPFN 2.5's feature wise attention heads distribute computation across layers. Using activation patching, ablation, and attention entropy across two synthetic regression datasets, we find clear temporal specialisation: one head's causal necessity dominates that of the others by 2 to 5 times at peak layer, with its dominant layer shifting across tasks of different complexity, while the remaining heads exhibit symmetric late layer profiles. Attention entropy and patching provide convergent evidence for the computationally active layers of the dominant head. We additionally investigate inference time steerability via contrastive activation steering, which fails to transfer across samples. We attribute this result to TabPFN's in context learning mechanism, which encodes task structure through context dependent attention rather than the stable parametric directions that make steering tractable in language models.