# Prior-Fitted Networks Scale to Larger Datasets When Treated as Weak Learners

arXiv: https://arxiv.org/abs/2503.01256 · HTML: https://mlsanity.com/paper/2503.01256/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 60 references that are in the map.
- First posted on arXiv 3 Mar 2025.

## Builds on (most cited first, 30 of 30)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data](https://mlsanity.com/paper/2003.06505/) (2020)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [Document Expansion by Query Prediction](https://mlsanity.com/paper/1904.08375/) (2019)
- [OpenML Benchmarking Suites](https://mlsanity.com/paper/1708.03731/) (2017)
- [Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning](https://mlsanity.com/paper/2007.04074/) (2020)
- [Well-tuned Simple Nets Excel on Tabular Datasets](https://mlsanity.com/paper/2106.11189/) (2021)
- [Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning](https://mlsanity.com/paper/2106.02584/) (2021)
- [Star-Transformer](https://mlsanity.com/paper/1902.09113/) (2019)
- [Statistical Foundations of Prior-Data Fitted Networks](https://mlsanity.com/paper/2305.11097/) (2023)
- [Trust Issues: Uncertainty Estimation Does Not Enable Reliable OOD Detection On Medical Tabular Data](https://mlsanity.com/paper/2011.03274/) (2020)
- [Sequential Deep Learning for Credit Risk Monitoring with Tabular Financial Data](https://mlsanity.com/paper/2012.15330/) (2020)
- [PromptBoosting: Black-Box Text Classification with Ten Forward Passes](https://mlsanity.com/paper/2212.09257/) (2022)
- [Applications and Techniques for Fast Machine Learning in Science](https://mlsanity.com/paper/2110.13041/) (2021)
- [LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrieval](https://mlsanity.com/paper/2203.06169/) (2022)
- [DC-BERT: Decoupling Question and Document for Efficient Contextual Encoding](https://mlsanity.com/paper/2002.12591/) (2020)
- [Google Dataset Search by the Numbers](https://mlsanity.com/paper/2006.06894/) (2020)
- [Randomized Gradient Boosting Machine](https://mlsanity.com/paper/1810.10158/) (2018)
- [What Dense Graph Do You Need for Self-Attention?](https://mlsanity.com/paper/2205.14014/) (2022)

## Built on by (most cited first, 7 of 7)

- [Representation Learning for Tabular Data: A Comprehensive Survey](https://mlsanity.com/paper/2504.16109/) (2025)
- [Effortless, Simulation-Efficient Bayesian Inference using Tabular Foundation Models](https://mlsanity.com/paper/2504.17660/) (2025)
- [From Zero to Hero: Advancing Zero-Shot Foundation Models for Tabular Outlier Detection](https://mlsanity.com/paper/2602.03018/) (2026)
- [Decoupled-Value Attention for Prior-Data Fitted Networks: GP Inference for Physical Equations](https://mlsanity.com/paper/2509.20950/) (2025)
- [uLEAD-TabPFN: Uncertainty-aware Dependency-based Anomaly Detection with TabPFN](https://mlsanity.com/paper/2604.20255/) (2026)
- [Transformers Can Learn Posterior Predictive Distributions In-Context](https://mlsanity.com/paper/2605.26713/) (2026)
- [Bayesian Wind Tunnels for Model Selection](https://mlsanity.com/paper/2607.19379/) (2026)

## Abstract (arXiv)

Prior-Fitted Networks (PFNs) have recently been proposed to efficiently perform tabular classification tasks. Although they achieve good performance on small datasets, they encounter limitations with larger datasets. These limitations include significant memory consumption and increased computational complexity, primarily due to the impracticality of incorporating all training samples as inputs within these networks. To address these challenges, we investigate the fitting assumption for PFNs and input samples. Building on this understanding, we propose textitBoostPFN designed to enhance the performance of these networks, especially for large-scale datasets. We also theoretically validate the convergence of BoostPFN and our empirical results demonstrate that the BoostPFN method can outperform standard PFNs with the same size of training samples in large datasets and achieve a significant acceleration in training times compared to other established baselines in the field, including widely-used Gradient Boosting Decision Trees (GBDTs), deep learning methods and AutoML systems. High performance is maintained for up to 50x of the pre-training size of PFNs, substantially extending the limit of training samples. Through this work, we address the challenges of efficiently handling large datasets via PFN-based models, paving the way for faster and more effective tabular data classification training and prediction process. Code is available at Github.