# Exploring Fine-Tuning for Tabular Foundation Models

arXiv: https://arxiv.org/abs/2601.09654 · HTML: https://mlsanity.com/paper/2601.09654/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 31 references that are in the map.
- First posted on arXiv 14 Jan 2026.

## Builds on (most cited first, 21 of 21)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [On Fairness and Calibration](https://mlsanity.com/paper/1709.02012/) (2017)
- [AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data](https://mlsanity.com/paper/2003.06505/) (2020)
- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [When Do Neural Nets Outperform Boosted Trees on Tabular Data?](https://mlsanity.com/paper/2305.02997/) (2023)
- [OpenML Benchmarking Suites](https://mlsanity.com/paper/1708.03731/) (2017)
- [On the Effectiveness of Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2211.15583/) (2022)
- [Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Models](https://mlsanity.com/paper/2510.21204/) (2025)
- [TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023](https://mlsanity.com/paper/2307.14338/) (2023)
- [Retrieval & Fine-Tuning for In-Context Tabular Models](https://mlsanity.com/paper/2406.05207/) (2024)
- [A Closer Look at Deep Learning Methods on Tabular Datasets](https://mlsanity.com/paper/2407.00956/) (2024)
- [Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learning](https://mlsanity.com/paper/2511.02818/) (2025)
- [A Survey on Deep Tabular Learning](https://mlsanity.com/paper/2410.12034/) (2024)
- [TabTune: A Unified Library for Inference and Fine-Tuning Tabular Foundation Models](https://mlsanity.com/paper/2511.02802/) (2025)
- [Orion-Bix: Bi-Axial Attention for Tabular In-Context Learning](https://mlsanity.com/paper/2512.00181/) (2025)

## Built on by (most cited first, 9 of 9)

- [Noise Immunity in In-Context Tabular Learning: An Empirical Robustness Analysis of TabPFN's Attention Mechanisms](https://mlsanity.com/paper/2604.04868/) (2026)
- [Shaping the Prior: How Synthetic Task Distributions Determine Tabular Foundation Model Quality](https://mlsanity.com/paper/2605.18971/) (2026)
- [TACTICL: Task-Aware Compression of Tabular ICL Models](https://mlsanity.com/paper/2608.10837/) (2026)
- [Distributional Regression with Tabular Foundation Models: Evaluating Probabilistic Predictions via Proper Scoring Rules](https://mlsanity.com/paper/2603.08206/) (2026)
- [Distilling Tabular Foundation Models for Structured Health Data](https://mlsanity.com/paper/2605.18702/) (2026)
- [TFM-Retouche: A Lightweight Input-Space Adapter for Tabular Foundation Models](https://mlsanity.com/paper/2605.06047/) (2026)
- [Data Presentation Over Architecture: Resampling Strategies for Credit Risk Prediction with Tabular Foundation Models](https://mlsanity.com/paper/2605.18635/) (2026)
- [Ensembling Tabular Foundation Models - A Diversity Ceiling And A Calibration Trap](https://mlsanity.com/paper/2605.18696/) (2026)
- [Topological Signatures of Context-Level Reliability in TabPFN](https://mlsanity.com/paper/2607.17962/) (2026)

## Abstract (arXiv)

Tabular Foundation Models (TFMs) have recently shown strong in-context learning capabilities on structured data, achieving zero-shot performance comparable to traditional machine learning methods. We find that zero-shot TFMs already achieve strong performance, while the benefits of fine-tuning are highly model and data-dependent. Meta-learning and PEFT provide moderate gains under specific conditions, whereas full supervised fine-tuning (SFT) often reduces accuracy or calibration quality. This work presents the first comprehensive study of fine-tuning in TFMs across benchmarks including TALENT, OpenML-CC18, and TabZilla. We compare Zero-Shot, Meta-Learning, Supervised (SFT), and parameter-efficient (PEFT) approaches, analyzing how dataset factors such as imbalance, size, and dimensionality affect outcomes. Our findings cover performance, calibration, and fairness, offering practical guidelines on when fine-tuning is most beneficial and its limitations.