mlsanityData to 14 Oct 2026

Real-TabPFN: Improving Tabular Foundation Models via Continued Pre-training With Real-World Data

arXiv 2507.03971 ↗First posted 5 Jul 2025cs.LG cs.AI stat.MEcited by 41

Not on the map. Why?

Abstract (arXiv)

Foundation models for tabular data, like TabPFN, achieve strong performance on small datasets when pre-trained solely on synthetic data. We show that this performance can be significantly boosted by a targeted continued pre-training phase. Specifically, we demonstrate that leveraging a small, curated collection of large, real-world datasets for continued pre-training yields superior downstream predictive accuracy compared to using broader, potentially noisier corpora like CommonCrawl or GitTables. Our resulting model, Real-TabPFN, achieves substantial performance gains on 29 datasets from the OpenML AutoML Benchmark.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Builds on 20 of its 31 references that are in the map. First posted on arXiv 5 Jul 2025.

Builds on

20 of 31 references.

References · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  6. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  7. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  8. 2014OpenML: networked science in machine learningcited by 526
  9. 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 398
  10. 2025TabICL: A Tabular Foundation Model for In-Context Learning on Large Datacited by 289
  11. 2018Explicit Inductive Bias for Transfer Learning with Convolutional Networkscited by 227
  12. 2022Why do tree-based models still outperform deep learning on tabular data?cited by 199
8 more
  1. 2024Investigating Data Contamination for Pre-training Language Modelscited by 91
  2. 2024Large Scale Transfer Learning for Tabular Data via Language Modelingcited by 76
  3. 2021GitTables: A Large-Scale Corpus of Relational Tablescited by 45
  4. 2017CatBoost: unbiased boosting with categorical featurescited by 31
  5. 2024Fine-tuned In-Context Learning Transformers are Excellent Tabular Data Classifierscited by 24
  6. 2023TabLib: A Dataset of 627M Tables with Contextcited by 23
  7. 2024Synthetic Tabular Data Generation for Imbalanced Classification: The Surprising Effectiveness of an Overlap Classcited by 8
  8. 2024TabularFM: An Open Framework For Tabular Foundational Modelscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 37.

41 papers.

Sort:

By area:

  1. 2025TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Modelscited by 132
  2. 2026KumoRFM-2: Scaling Foundation Models for Relational Learningcited by 19
  3. 2025TabPFN-Wide: Continued Pre-Training for Extreme Feature Countscited by 16
  4. 2026ScoringBench: A Benchmark for Evaluating Tabular Foundation Models with Proper Scoring Rulescited by 6
  5. 2026Tabular foundation models for in-context prediction of molecular propertiescited by 6
  6. 2026TabFM: A Zero-Shot Foundation Model for Tabular Datacited by 6
  7. 2025Learning-To-Measure: In-Context Active Feature Acquisitioncited by 5
  8. 2026MacrOData: New Benchmarks of Thousands of Datasets for Tabular Outlier Detectioncited by 5
  9. 2026Interpretable Tabular Foundation Models via In-Context Kernel Regressioncited by 4
  10. 2026Retrieval-aligned Tabular Foundation Models Enable Robust Clinical Risk Prediction in Electronic Health Records Under Real-world Constraintscited by 4
  11. 2026Towards Evaluating Data Priors for Tabular Foundation Modelscited by 4
  12. 2026TACTICL: Task-Aware Compression of Tabular ICL Modelscited by 4
29 more
  1. 2026Causal Data Augmentation for Robust Fine-Tuning of Tabular Foundation Modelscited by 3
  2. 2026Engineering Regression Without Real-Data Training: Domain Adaptation for Tabular Foundation Models Using Multi-Dataset Embeddingscited by 3
  3. 2026FIRE: Multi-fidelity Regression with Distribution-conditioned In-context Learning using Tabular Foundation Modelscited by 2
  4. 2026Rational ANOVA Networkscited by 2
  5. 2026FoMo X: Modular Explainability Signals for Outlier Detection Foundation Modelscited by 2
  6. 2026MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Imagecited by 2
  7. 2026Efficient Adaptive Data Acquisition via Pretrained Belief Representationscited by 2
  8. 2026Understanding the Surprising Generalization Properties of Tabular Foundation Modelscited by 2
  9. 2026SBBTS: A Unified Schr"odinger-Bass Framework for Synthetic Financial Time Seriescited by 1
  10. 2026Breaking the Quality-Privacy Tradeoff in Tabular Data Generation via In-Context Learningcited by 1
  11. 2026When Tabular Foundation Models Meet Strategic Tabular Data: A Prior Alignment Approachcited by 1
  12. 2026Explaining Tabular Foundation Model Differences Through Meta-Featurescited by 1
  13. 2026Probing Memorization of Tabular In-Context Learningcited by 1
  14. 2026Causal Foundation Modelscited by 1
  15. 2026From Synthetic Priors to Model Behavior: Structural Coverage in Tabular Foundation Modelscited by 1
  16. 2025An Empirical Study of Machine Learning Robustness and Scalability for Imbalanced Tabular Clinical Data in Emergency and Critical Carecited by 0
  17. 2026Ensembling Tabular Foundation Models - A Diversity Ceiling And A Calibration Trapcited by 0
  18. 2026In-Context Multiple Instance Learningcited by 0
  19. 2026Simultaneous hyperkinetic movement disorders phenotyping: a cross-cohort pediatric transfer study using routine videos, markerless pose estimation and a tabular foundation modelcited by 0
  20. 2026In-Context Learning for Latent Space Bayesian Optimizationcited by 0
  21. 2026From field-scale to large-scale spectral libraries: Tabular foundation models in soil spectroscopycited by 0
  22. 2026Training Fair Tabular Foundation Modelscited by 0
  23. 2026Do Tabular Foundation Models Know Physics? Contamination, Units, and the Deterministic Limitcited by 0
  24. 2026TabBench-Bio: A Living Benchmark for Machine Learning on High-Dimensional Biomedical Tablescited by 0
  25. 2026TabFM-Auto: Self-Evolving Pipelines for Tabular Foundation Modelscited by 0
  26. 2026TAFFY: A Task-Adaptive Tabular Foundation Model with In-Context Diversitycited by 0
  27. 2026GIST: Generator-state-Informed Supervision for Tabular Foundation Modelscited by 0
  28. 2026How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?cited by 0
  29. 2026When Does Depth Help? Understanding Layerwise Inference in Tabular Foundation Modelscited by 0