# Benchmarking Distribution Shift in Tabular Data with TableShift

arXiv: https://arxiv.org/abs/2312.07577 · HTML: https://mlsanity.com/paper/2312.07577/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 90 references that are in the map.
- First posted on arXiv 10 Dec 2023.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Deep CORAL: Correlation Alignment for Deep Domain Adaptation](https://mlsanity.com/paper/1607.01719/) (2016)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Out-of-Distribution Generalization via Risk Extrapolation (REx)](https://mlsanity.com/paper/2003.00688/) (2020)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [Retiring Adult: New Datasets for Fair Machine Learning](https://mlsanity.com/paper/2108.04884/) (2021)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [Multitask learning and benchmarking with clinical time series data](https://mlsanity.com/paper/1703.07771/) (2017)
- [Adversarial Domain Adaptation with Domain Mixup](https://mlsanity.com/paper/1912.01805/) (2019)
- [Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization](https://mlsanity.com/paper/2107.04649/) (2021)
- [A comparative study of fairness-enhancing interventions in machine learning](https://mlsanity.com/paper/1802.04422/) (2018)
- [CatBoost: gradient boosting with categorical features support](https://mlsanity.com/paper/1810.11363/) (2018)
- [Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data](https://mlsanity.com/paper/1909.06312/) (2019)
- [Why do tree-based models still outperform deep learning on tabular data?](https://mlsanity.com/paper/2207.08815/) (2022)
- [Improve Unsupervised Domain Adaptation with Mixup Training](https://mlsanity.com/paper/2001.00677/) (2020)
- [Large-Scale Methods for Distributionally Robust Optimization](https://mlsanity.com/paper/2010.05893/) (2020)
- [Geometric Dataset Distances via Optimal Transport](https://mlsanity.com/paper/2002.02923/) (2020)
- [Well-tuned Simple Nets Excel on Tabular Datasets](https://mlsanity.com/paper/2106.11189/) (2021)
- [MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-III](https://mlsanity.com/paper/1907.08322/) (2019)
- [The Effect of Natural Distribution Shift on Question Answering Models](https://mlsanity.com/paper/2004.14444/) (2020)
- [Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks](https://mlsanity.com/paper/2107.07455/) (2021)

## Built on by (most cited first, 30 of 66)

- [Why Tabular Foundation Models Should Be a Research Priority](https://mlsanity.com/paper/2405.01147/) (2024)
- [Large Scale Transfer Learning for Tabular Data via Language Modeling](https://mlsanity.com/paper/2406.12031/) (2024)
- [Representation Learning for Tabular Data: A Comprehensive Survey](https://mlsanity.com/paper/2504.16109/) (2025)
- [TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarks](https://mlsanity.com/paper/2406.19380/) (2024)
- [Drift-Resilient TabPFN: In-Context Learning Temporal Distribution Shifts on Tabular Data](https://mlsanity.com/paper/2411.10634/) (2024)
- [BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Model](https://mlsanity.com/paper/2404.03830/) (2024)
- [The Landscape of Causal Discovery Data: Grounding Causal Discovery in Real-World Applications](https://mlsanity.com/paper/2412.01953/) (2024)
- [Text Serialization and Their Relationship with the Conventional Paradigms of Tabular Machine Learning](https://mlsanity.com/paper/2406.13846/) (2024)
- [A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Data](https://mlsanity.com/paper/2407.02112/) (2024)
- [TabularBench: Benchmarking Adversarial Robustness for Tabular Deep Learning in Real-world Use-cases](https://mlsanity.com/paper/2408.07579/) (2024)
- [Estimating Model Performance Under Covariate Shift Without Labels](https://mlsanity.com/paper/2401.08348/) (2024)
- [Causality for Tabular Data Synthesis: A High-Order Structure Causal Benchmark Framework](https://mlsanity.com/paper/2406.08311/) (2024)
- [Confidence-based Estimators for Predictive Performance in Model Monitoring](https://mlsanity.com/paper/2407.08649/) (2024)
- [ABCFair: an Adaptable Benchmark approach for Comparing Fairness Methods](https://mlsanity.com/paper/2409.16965/) (2024)
- [Fully Test-time Adaptation for Tabular Data](https://mlsanity.com/paper/2412.10871/) (2024)
- [Towards Fair In-Context Learning with Tabular Foundation Models](https://mlsanity.com/paper/2505.09503/) (2025)
- [CLIMB: Class-imbalanced Learning Benchmark on Tabular Data](https://mlsanity.com/paper/2505.17451/) (2025)
- [SIMSHIFT: A Benchmark for Adapting Neural Surrogates to Distribution Shifts](https://mlsanity.com/paper/2506.12007/) (2025)
- [Quantifying Prediction Consistency Under Fine-Tuning Multiplicity in Tabular LLMs](https://mlsanity.com/paper/2407.04173/) (2024)
- [AdapTable: Test-Time Adaptation for Tabular Data via Shift-Aware Uncertainty Calibrator and Label Distribution Handler](https://mlsanity.com/paper/2407.10784/) (2024)
- [Multi-Source and Test-Time Domain Adaptation on Multivariate Signals using Spatio-Temporal Monge Alignment](https://mlsanity.com/paper/2407.14303/) (2024)
- [Keeping Medical AI Healthy and Trustworthy: A Review of Detection and Correction Methods for System Degradation](https://mlsanity.com/paper/2506.17442/) (2025)
- [End-to-End Compression for Tabular Foundation Models](https://mlsanity.com/paper/2602.05649/) (2026)
- [Understanding the Limits of Deep Tabular Methods with Temporal Shift](https://mlsanity.com/paper/2502.20260/) (2025)
- [When Shift Happens - Confounding Is to Blame](https://mlsanity.com/paper/2505.21422/) (2025)
- [DRO: A Python Library for Distributionally Robust Optimization in Machine Learning](https://mlsanity.com/paper/2505.23565/) (2025)
- [Realistic Evaluation of TabPFN v2 in Open Environments](https://mlsanity.com/paper/2505.16226/) (2025)
- [Mitigating Label Shift in Tabular In-Context Learning via Test-Time Posterior Adjustment](https://mlsanity.com/paper/2605.04363/) (2026)
- [LakeMLB: Data Lake Machine Learning Benchmark](https://mlsanity.com/paper/2602.10441/) (2026)
- [SKADA-Bench: Benchmarking Unsupervised Domain Adaptation Methods with Realistic Validation On Diverse Modalities](https://mlsanity.com/paper/2407.11676/) (2024)

## Abstract (arXiv)

Robustness to distribution shift has become a growing concern for text and image models as they transition from research subjects to deployment in the real world. However, high-quality benchmarks for distribution shift in tabular machine learning tasks are still lacking despite the widespread real-world use of tabular data and differences in the models used for tabular data in comparison to text and images. As a consequence, the robustness of tabular models to distribution shift is poorly understood. To address this issue, we introduce TableShift, a distribution shift benchmark for tabular data. TableShift contains 15 binary classification tasks in total, each with an associated shift, and includes a diverse set of data sources, prediction targets, and distribution shifts. The benchmark covers domains including finance, education, public policy, healthcare, and civic participation, and is accessible using only a few lines of Python code via the TableShift API. We conduct a large-scale study comparing several state-of-the-art tabular data models alongside robust learning and domain generalization methods on the benchmark tasks. Our study demonstrates (1) a linear trend between in-distribution (ID) and out-of-distribution (OOD) accuracy; (2) domain robustness methods can reduce shift gaps but at the cost of reduced ID accuracy; (3) a strong relationship between shift gap (difference between ID and OOD performance) and shifts in the label distribution. The benchmark data, Python package, model implementations, and more information about TableShift are available at https://github.com/mlfoundations/tableshift and https://tableshift.org .