# Towards Understanding Layer Contributions in Tabular In-Context Learning Models

arXiv: https://arxiv.org/abs/2511.15432 · HTML: https://mlsanity.com/paper/2511.15432/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 16 references that are in the map.
- First posted on arXiv 19 Nov 2025.

## Builds on (most cited first, 14 of 14)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018)
- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [TabArena: A Living Benchmark for Machine Learning on Tabular Data](https://mlsanity.com/paper/2506.16791/) (2025)
- [Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation](https://mlsanity.com/paper/1810.11750/) (2018)
- [Transformer Layers as Painters](https://mlsanity.com/paper/2407.09298/) (2024)
- [A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities](https://mlsanity.com/paper/2502.17361/) (2025)
- [Investigating Layer Importance in Large Language Models](https://mlsanity.com/paper/2409.14381/) (2024)
- [Skip a Layer or Loop it? Test-Time Depth Adaptation of Pretrained LLMs](https://mlsanity.com/paper/2507.07996/) (2025)
- [Early Stopping Tabular In-Context Learning](https://mlsanity.com/paper/2506.21387/) (2025)
- [Dr.LLM: Dynamic Layer Routing in LLMs](https://mlsanity.com/paper/2510.12773/) (2025)
- [What Layers When: Learning to Skip Compute in LLMs with Residual Gates](https://mlsanity.com/paper/2510.13876/) (2025)

## Built on by (most cited first, 1 of 1)

- [TACTICL: Task-Aware Compression of Tabular ICL Models](https://mlsanity.com/paper/2608.10837/) (2026)

## Abstract (arXiv)

Despite the architectural similarities between tabular in-context learning (ICL) models and large language models (LLMs), little is known about how individual layers contribute to tabular prediction. In this paper, we investigate how the latent spaces evolve across layers in tabular ICL models, identify potential redundant layers, and compare these dynamics with those observed in LLMs. We analyze TabPFN and TabICL through the "layers as painters" perspective, finding that only subsets of layers share a common representational language, suggesting structural redundancy and offering opportunities for model compression and improved interpretability.