# TabPFN Through The Looking Glass: An interpretability study of TabPFN and its internal representations

arXiv: https://arxiv.org/abs/2601.08181 · HTML: https://mlsanity.com/paper/2601.08181/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 29 references that are in the map.
- First posted on arXiv 13 Jan 2026.

## Builds on (most cited first, 10 of 10)

- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023)
- [Why do tree-based models still outperform deep learning on tabular data?](https://mlsanity.com/paper/2207.08815/) (2022)
- [A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities](https://mlsanity.com/paper/2502.17361/) (2025)
- [Exploring Representations and Interventions in Time Series Foundation Models](https://mlsanity.com/paper/2409.12915/) (2024)
- [Enhancing Actuarial Non-Life Pricing Models via Transformers](https://mlsanity.com/paper/2311.07597/) (2023)

## Built on by (most cited first, 2 of 2)

- [Pre-trained Tabular Foundation Models as Versatile Summary Networks for Neural Posterior Estimation](https://mlsanity.com/paper/2605.07765/) (2026)
- [The Standardization Trap: Certifying Joint Label Processing in Tabular Foundation Models](https://mlsanity.com/paper/2610.08314/) (2026)

## Abstract (arXiv)

Tabular foundational models are pre-trained models designed for a wide range of tabular data tasks. They have shown strong performance across domains, yet their internal representations and learned concepts remain poorly understood. This lack of interpretability makes it important to study how these models process and transform input features. In this work, we analyze the information encoded inside the model's hidden representations and examine how these representations evolve across layers. We run a set of probing experiments that test for the presence of linear regression coefficients, intermediate values from complex expressions, and the final answer in early layers. These experiments allow us to reason about the computations the model performs internally. Our results provide evidence that meaningful and structured information is stored inside the representations of tabular foundational models. We observe clear signals that correspond to both intermediate and final quantities involved in the model's prediction process. This gives insight into how the model refines its inputs and how the final output emerges. Our findings contribute to a deeper understanding of the internal mechanics of tabular foundational models. They show that these models encode concrete and interpretable information, which moves us closer to making their decision processes more transparent and trustworthy.