# Characterizing the Separability Gap in Prior-Data Fitted Networks

arXiv: https://arxiv.org/abs/2610.14989 · HTML: https://mlsanity.com/paper/2610.14989/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 42 references that are in the map.
- First posted on arXiv 10 Oct 2026.

## Builds on (most cited first, 28 of 28)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [Conditional Neural Processes](https://mlsanity.com/paper/1807.01613/) (2018)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [Neural Processes](https://mlsanity.com/paper/1807.01622/) (2018)
- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [Attentive Neural Processes](https://mlsanity.com/paper/1901.05761/) (2019)
- [Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models](https://mlsanity.com/paper/1612.08468/) (2016)
- [TabArena: A Living Benchmark for Machine Learning on Tabular Data](https://mlsanity.com/paper/2506.16791/) (2025)
- [Feature relevance quantification in explainable AI: A causal problem](https://mlsanity.com/paper/1910.13413/) (2019)
- [TabICLv2: A better, faster, scalable, and open tabular foundation model](https://mlsanity.com/paper/2602.11139/) (2026)
- [Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modeling](https://mlsanity.com/paper/2207.04179/) (2022)
- [LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligence](https://mlsanity.com/paper/2509.03505/) (2025)
- [Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Models](https://mlsanity.com/paper/2510.21204/) (2025)
- [Meta-Learning Stationary Stochastic Process Prediction with Convolutional Neural Processes](https://mlsanity.com/paper/2007.01332/) (2020)
- [PFNs4BO: In-Context Learning for Bayesian Optimization](https://mlsanity.com/paper/2305.17535/) (2023)
- [Statistical Foundations of Prior-Data Fitted Networks](https://mlsanity.com/paper/2305.11097/) (2023)
- [Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspective](https://mlsanity.com/paper/2406.00793/) (2024)
- [TALENT: A Tabular Analytics and Learning Toolbox](https://mlsanity.com/paper/2407.04057/) (2024)
- [Martingale posterior distributions](https://mlsanity.com/paper/2103.15671/) (2021)
- [Beyond IID: How General Are Tabular Foundation Models, Really?](https://mlsanity.com/paper/2606.30410/) (2026)
- [Interpretable Machine Learning for TabPFN](https://mlsanity.com/paper/2403.10923/) (2024)
- [TabMGP: Martingale Posterior with TabPFN](https://mlsanity.com/paper/2510.25154/) (2025)
- [Bayes' Power for Explaining In-Context Learning Generalizations](https://mlsanity.com/paper/2410.01565/) (2024)
- [When and How Unlabeled Data Provably Improve In-Context Learning](https://mlsanity.com/paper/2506.15329/) (2025)
- [ScoringBench: A Benchmark for Evaluating Tabular Foundation Models with Proper Scoring Rules](https://mlsanity.com/paper/2603.29928/) (2026)
- [Does TabPFN Understand Causal Structures?](https://mlsanity.com/paper/2511.07236/) (2025)

## Abstract (arXiv)

Prior-data fitted networks (PFNs) are transformers pretrained on simulated datasets to approximate Bayesian posterior predictive distributions. Standard PFN theory assumes a separable task prior, under which the feature distribution and the mechanism generating labels from features are independent. We show that common PFN priors are non-separable, even in simple settings. Consequently, standard pretraining targets a predictive distribution based on a task posterior updated using both the labeled context and the query features. We characterize its difference from the predictive based on the context posterior alone and show that this difference vanishes under posterior concentration. Under misspecification (i.e., when the true joint distribution lies outside the prior's support), however, prior non-separability can favor better feature fit at the expense of conditional label fit. We illustrate the consequences for feature attribution, Bayesian optimization, and martingale checks of Bayesian behavior. The same dependence also enables PFNs to learn from additional unlabeled observations. We demonstrate this with SemiTabICL, a PFN explicitly pretrained to use these observations, which improves accuracy over a matched supervised baseline. Our findings highlight the need for explicit rules on unlabeled data use in PFN evaluation and establish the dependence between feature and label distributions as a consequential prior design choice.