# Shaping the Prior: How Synthetic Task Distributions Determine Tabular Foundation Model Quality

arXiv: https://arxiv.org/abs/2605.18971 · HTML: https://mlsanity.com/paper/2605.18971/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 36 references that are in the map.
- First posted on arXiv 18 May 2026.

## Builds on (most cited first, 23 of 23)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Modeling Tabular data using Conditional GAN](https://mlsanity.com/paper/1907.00503/) (2019)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [From Louvain to Leiden: guaranteeing well-connected communities](https://mlsanity.com/paper/1810.08473/) (2018)
- [TabDDPM: Modelling Tabular Data with Diffusion Models](https://mlsanity.com/paper/2209.15421/) (2022)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [Language Models are Realistic Tabular Data Generators](https://mlsanity.com/paper/2210.06280/) (2022)
- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [OpenML Benchmarking Suites](https://mlsanity.com/paper/1708.03731/) (2017)
- [TabArena: A Living Benchmark for Machine Learning on Tabular Data](https://mlsanity.com/paper/2506.16791/) (2025)
- [TabICLv2: A better, faster, scalable, and open tabular foundation model](https://mlsanity.com/paper/2602.11139/) (2026)
- [Foundations of Structural Causal Models with Cycles and Latent Variables](https://mlsanity.com/paper/1611.06221/) (2016)
- [TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models](https://mlsanity.com/paper/2511.08667/) (2025)
- [LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligence](https://mlsanity.com/paper/2509.03505/) (2025)
- [Fine-tuned In-Context Learning Transformers are Excellent Tabular Data Classifiers](https://mlsanity.com/paper/2405.13396/) (2024)
- [Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learning](https://mlsanity.com/paper/2511.02818/) (2025)
- [nanoTabPFN: A Lightweight and Educational Reimplementation of TabPFN](https://mlsanity.com/paper/2511.03634/) (2025)
- [Relational In-Context Learning via Synthetic Pre-training with Structural Prior](https://mlsanity.com/paper/2603.03805/) (2026)
- [TabTune: A Unified Library for Inference and Fine-Tuning Tabular Foundation Models](https://mlsanity.com/paper/2511.02802/) (2025)
- [Orion-Bix: Bi-Axial Attention for Tabular In-Context Learning](https://mlsanity.com/paper/2512.00181/) (2025)
- [TabStruct: Measuring Structural Fidelity of Tabular Data](https://mlsanity.com/paper/2509.11950/) (2025)
- [Exploring Fine-Tuning for Tabular Foundation Models](https://mlsanity.com/paper/2601.09654/) (2026)
- [Improving TabPFN's Synthetic Data Generation by Integrating Causal Structure](https://mlsanity.com/paper/2603.10254/) (2026)

## Built on by (most cited first, 6 of 6)

- [Mitra-v2 Technical Report](https://mlsanity.com/paper/2609.04540/) (2026)
- [A JEPA Recipe for Tabular Foundation Models](https://mlsanity.com/paper/2609.25541/) (2026)
- [From Behavior to Provenance: Attributing Tabular Foundation Models to Synthetic Pretraining Data](https://mlsanity.com/paper/2610.02347/) (2026)
- [When Does Synthetic Relational Data Teach Models to Use Relations? Tracing Predictive Structure from Pretraining Data to Model Behavior](https://mlsanity.com/paper/2610.03057/) (2026)
- [TAFFY: A Task-Adaptive Tabular Foundation Model with In-Context Diversity](https://mlsanity.com/paper/2610.07559/) (2026)
- [OrionMoE-v1.0: A Mixture-of-Experts Foundation Model for Tabular In-Context Learning](https://mlsanity.com/paper/2610.20418/) (2026)

## Abstract (arXiv)

What determines the quality of a tabular foundation model? Unlike language or vision, tabular foundation models acquire their inductive biases almost entirely from synthetic pretraining distributions, yet the design of these distributions remains poorly understood. Standard synthetic priors are too well-behaved: they omit the irregularities and failure modes that determine deployment robustness. We introduce O'Prior, a compositional realism prior built around four coupled components: a hierarchical SCM meta-generator spanning diverse functional families; a modular realism engine covering heterogeneous marginals, missingness, and target transforms; an explicit stress module injecting confounding and support-query mismatch; and a curriculum-governed, leakage-safe generation protocol. To isolate prior design as the scientific variable, we hold architecture, optimizer, and compute budget fixed and vary only the synthetic task distribution. O'Prior yields consistent and substantial improvements in downstream accuracy and robustness across real tabular benchmarks, with gains concentrated in regimes characterized by distributional irregularities. Ablations confirm that mechanism diversity, realism composition, and shift-aware stress each contribute independently, their effects are not interchangeable. These results establish synthetic prior construction as a first-order and largely overlooked determinant of tabular foundation model quality