# Fine-tuned In-Context Learning Transformers are Excellent Tabular Data Classifiers

arXiv: https://arxiv.org/abs/2405.13396 · HTML: https://mlsanity.com/paper/2405.13396/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 May 2024.

## Built on by (most cited first, 24 of 24)

- [TabDPT: Scaling Tabular Foundation Models on Real Data](https://mlsanity.com/paper/2410.18164/) (2024)
- [Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Models](https://mlsanity.com/paper/2510.21204/) (2025)
- [A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities](https://mlsanity.com/paper/2502.17361/) (2025)
- [Real-TabPFN: Improving Tabular Foundation Models via Continued Pre-training With Real-World Data](https://mlsanity.com/paper/2507.03971/) (2025)
- [Representation Learning for Tabular Data: A Comprehensive Survey](https://mlsanity.com/paper/2504.16109/) (2025)
- [Table Foundation Models: on knowledge pre-training for tabular learning](https://mlsanity.com/paper/2505.14415/) (2025)
- [PluRel: Synthetic Data unlocks Scaling Laws for Relational Foundation Models](https://mlsanity.com/paper/2602.04029/) (2026)
- [KumoRFM-2: Scaling Foundation Models for Relational Learning](https://mlsanity.com/paper/2604.12596/) (2026)
- [TabPFN-Wide: Continued Pre-Training for Extreme Feature Counts](https://mlsanity.com/paper/2510.06162/) (2025)
- [TACTIC for Navigating the Unknown: Tabular Anomaly deteCTion via In-Context inference](https://mlsanity.com/paper/2603.14171/) (2026)
- [Shaping the Prior: How Synthetic Task Distributions Determine Tabular Foundation Model Quality](https://mlsanity.com/paper/2605.18971/) (2026)
- [Unmasking Trees for Tabular Data](https://mlsanity.com/paper/2407.05593/) (2024)
- [Realistic Evaluation of TabPFN v2 in Open Environments](https://mlsanity.com/paper/2505.16226/) (2025)
- [Towards Evaluating Data Priors for Tabular Foundation Models](https://mlsanity.com/paper/2606.29241/) (2026)
- [On the Robustness of Tabular Foundation Models: Test-Time Attacks and In-Context Defenses](https://mlsanity.com/paper/2506.02978/) (2025)
- [Causal Data Augmentation for Robust Fine-Tuning of Tabular Foundation Models](https://mlsanity.com/paper/2601.04110/) (2026)
- [Mind the Gap? A Distributional Comparison of Real and Synthetic Priors for Tabular Foundation Models](https://mlsanity.com/paper/2605.06343/) (2026)
- [In-Context Decision Making for Optimizing Complex AutoML Pipelines](https://mlsanity.com/paper/2508.13657/) (2025)
- [CaDrift: A Time-dependent Causal Generator of Drifting Data Streams](https://mlsanity.com/paper/2602.20329/) (2026)
- [SeBA: Semi-supervised few-shot learning via Separated-at-Birth Alignment for tabular data](https://mlsanity.com/paper/2605.08519/) (2026)
- [Algorithmic Recourse of In-Context Learning for Tabular Data](https://mlsanity.com/paper/2605.31272/) (2026)
- [The Geometry of Logic: Stratification Induces Semantic Structure and Robust Reasoning](https://mlsanity.com/paper/2609.32927/) (2026)
- [TAFFY: A Task-Adaptive Tabular Foundation Model with In-Context Diversity](https://mlsanity.com/paper/2610.07559/) (2026)
- [How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?](https://mlsanity.com/paper/2610.13764/) (2026)

## Abstract (arXiv)

The recently introduced TabPFN pretrains an In-Context Learning (ICL) transformer on synthetic data to perform tabular data classification. In this work, we extend TabPFN to the fine-tuning setting, resulting in a significant performance boost. We also discover that fine-tuning enables ICL-transformers to create complex decision boundaries, a property regular neural networks do not have. Based on this observation, we propose to pretrain ICL-transformers on a new forest dataset generator which creates datasets that are unrealistic, but have complex decision boundaries. TabForest, the ICL-transformer pretrained on this dataset generator, shows better fine-tuning performance when pretrained on more complex datasets. Additionally, TabForest outperforms TabPFN on some real-world datasets when fine-tuning, despite having lower zero-shot performance due to the unrealistic nature of the pretraining datasets. By combining both dataset generators, we create TabForestPFN, an ICL-transformer that achieves excellent fine-tuning performance and good zero-shot performance.