mlsanityData to 14 Oct 2026

Exploring Fine-Tuning for Tabular Foundation Models

arXiv 2601.09654 ↗First posted 14 Jan 2026cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Tabular Foundation Models (TFMs) have recently shown strong in-context learning capabilities on structured data, achieving zero-shot performance comparable to traditional machine learning methods. We find that zero-shot TFMs already achieve strong performance, while the benefits of fine-tuning are highly model and data-dependent. Meta-learning and PEFT provide moderate gains under specific conditions, whereas full supervised fine-tuning (SFT) often reduces accuracy or calibration quality. This work presents the first comprehensive study of fine-tuning in TFMs across benchmarks including TALENT, OpenML-CC18, and TabZilla. We compare Zero-Shot, Meta-Learning, Supervised (SFT), and parameter-efficient (PEFT) approaches, analyzing how dataset factors such as imbalance, size, and dimensionality affect outcomes. Our findings cover performance, calibration, and fairness, offering practical guidelines on when fine-tuning is most beneficial and its limitations.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 21 of its 31 references that are in the map. First posted on arXiv 14 Jan 2026.

Builds on

21 of 31 references.

References · 21

  1. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  4. 2016Equality of Opportunity in Supervised Learningcited by 2,007
  5. 2021Revisiting Deep Learning Models for Tabular Datacited by 740
  6. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  7. 2021Deep Neural Networks and Tabular Data: A Surveycited by 491
  8. 2017On Fairness and Calibrationcited by 406
  9. 2020AutoGluon-Tabular: Robust and Accurate AutoML for Structured Datacited by 383
  10. 2025TabICL: A Tabular Foundation Model for In-Context Learning on Large Datacited by 289
  11. 2023When Do Neural Nets Outperform Boosted Trees on Tabular Data?cited by 223
  12. 2017OpenML Benchmarking Suitescited by 178
9 more
  1. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 141
  2. 2025Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Modelscited by 77
  3. 2023TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023cited by 74
  4. 2024Retrieval & Fine-Tuning for In-Context Tabular Modelscited by 59
  5. 2024A Closer Look at Deep Learning Methods on Tabular Datasetscited by 41
  6. 2025Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learningcited by 19
  7. 2024A Survey on Deep Tabular Learningcited by 18
  8. 2025TabTune: A Unified Library for Inference and Fine-Tuning Tabular Foundation Modelscited by 11
  9. 2025Orion-Bix: Bi-Axial Attention for Tabular In-Context Learningcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026Noise Immunity in In-Context Tabular Learning: An Empirical Robustness Analysis of TabPFN's Attention Mechanismscited by 6
  2. 2026Shaping the Prior: How Synthetic Task Distributions Determine Tabular Foundation Model Qualitycited by 6
  3. 2026TACTICL: Task-Aware Compression of Tabular ICL Modelscited by 4
  4. 2026Distributional Regression with Tabular Foundation Models: Evaluating Probabilistic Predictions via Proper Scoring Rulescited by 3
  5. 2026Distilling Tabular Foundation Models for Structured Health Datacited by 2
  6. 2026TFM-Retouche: A Lightweight Input-Space Adapter for Tabular Foundation Modelscited by 0
  7. 2026Data Presentation Over Architecture: Resampling Strategies for Credit Risk Prediction with Tabular Foundation Modelscited by 0
  8. 2026Ensembling Tabular Foundation Models - A Diversity Ceiling And A Calibration Trapcited by 0
  9. 2026Topological Signatures of Context-Level Reliability in TabPFNcited by 0