# Speedrunning Tabular Foundation Model Pretraining

arXiv: https://arxiv.org/abs/2606.03681 · HTML: https://mlsanity.com/paper/2606.03681/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 23 references that are in the map.
- First posted on arXiv 2 Jun 2026.

## Builds on (most cited first, 16 of 16)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [TabArena: A Living Benchmark for Machine Learning on Tabular Data](https://mlsanity.com/paper/2506.16791/) (2025)
- [The Road Less Scheduled](https://mlsanity.com/paper/2405.15682/) (2024)
- [TabICLv2: A better, faster, scalable, and open tabular foundation model](https://mlsanity.com/paper/2602.11139/) (2026)
- [TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models](https://mlsanity.com/paper/2511.08667/) (2025)
- [TabDPT: Scaling Tabular Foundation Models on Real Data](https://mlsanity.com/paper/2410.18164/) (2024)
- [Why Tabular Foundation Models Should Be a Research Priority](https://mlsanity.com/paper/2405.01147/) (2024)
- [Cramming: Training a Language Model on a Single GPU in One Day](https://mlsanity.com/paper/2212.14034/) (2022)
- [LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligence](https://mlsanity.com/paper/2509.03505/) (2025)
- [Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averaging](https://mlsanity.com/paper/2209.14981/) (2022)
- [Early Weight Averaging meets High Learning Rates for LLM Pre-training](https://mlsanity.com/paper/2306.03241/) (2023)

## Abstract (arXiv)

Pretraining cost is a major bottleneck for research on tabular foundation models, slowing the iteration cycle for new architectures, priors, and optimization ideas. Yet the community lacks a simple way to compare and accumulate pretraining speedups. We introduce a community speedrun for nanoTabPFN: contributors modify a single-file training script and compete to reach a fixed downstream ROC AUC target on subsampled TabArena using one NVIDIA L40S GPU. The current best record reaches the target in 0.92 minutes, an 81x speedup over the 74.32 minute baseline while using 22x fewer synthetic datasets. The speedrun format provides a simple protocol for the community to add, verify, and stack pretraining improvements, with the leaderboard open to contributions. Code and records are available at https://github.com/borawhocodess/modded-nanotabpfn.