mlsanityData to 7 Oct 2026

Benchmarking Distribution Shift in Tabular Data with TableShift

arXiv 2312.07577 ↗First posted 10 Dec 2023cs.LGcited by 64

Not on the map. Why?

Abstract (arXiv)

Robustness to distribution shift has become a growing concern for text and image models as they transition from research subjects to deployment in the real world. However, high-quality benchmarks for distribution shift in tabular machine learning tasks are still lacking despite the widespread real-world use of tabular data and differences in the models used for tabular data in comparison to text and images. As a consequence, the robustness of tabular models to distribution shift is poorly understood. To address this issue, we introduce TableShift, a distribution shift benchmark for tabular data. TableShift contains 15 binary classification tasks in total, each with an associated shift, and includes a diverse set of data sources, prediction targets, and distribution shifts. The benchmark covers domains including finance, education, public policy, healthcare, and civic participation, and is accessible using only a few lines of Python code via the TableShift API. We conduct a large-scale study comparing several state-of-the-art tabular data models alongside robust learning and domain generalization methods on the benchmark tasks. Our study demonstrates (1) a linear trend between in-distribution (ID) and out-of-distribution (OOD) accuracy; (2) domain robustness methods can reduce shift gaps but at the cost of reduced ID accuracy; (3) a strong relationship between shift gap (difference between ID and OOD performance) and shifts in the label distribution. The benchmark data, Python package, model implementations, and more information about TableShift are available at https://github.com/mlfoundations/tableshift and https://tableshift.org .

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Builds on 40 of its 90 references that are in the map. First posted on arXiv 10 Dec 2023.

Builds on

40 of 90 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  5. 2019Invariant Risk Minimizationcited by 1,832
  6. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  7. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  8. 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,194
  9. 2020In Search of Lost Domain Generalizationcited by 1,042
  10. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 737
  11. 2021Revisiting Deep Learning Models for Tabular Datacited by 725
  12. 2021Tabular Data: Deep Learning is Not All You Needcited by 530
28 more
  1. 2021Deep Neural Networks and Tabular Data: A Surveycited by 489
  2. 2021Retiring Adult: New Datasets for Fair Machine Learningcited by 443
  3. 2020TabTransformer: Tabular Data Modeling Using Contextual Embeddingscited by 398
  4. 2021SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Trainingcited by 310
  5. 2017Multitask learning and benchmarking with clinical time series datacited by 287
  6. 2019Adversarial Domain Adaptation with Domain Mixupcited by 250
  7. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 248
  8. 2018A comparative study of fairness-enhancing interventions in machine learningcited by 208
  9. 2018CatBoost: gradient boosting with categorical features supportcited by 208
  10. 2019Neural Oblivious Decision Ensembles for Deep Learning on Tabular Datacited by 200
  11. 2022Why do tree-based models still outperform deep learning on tabular data?cited by 198
  12. 2020Improve Unsupervised Domain Adaptation with Mixup Trainingcited by 185
  13. 2020Large-Scale Methods for Distributionally Robust Optimizationcited by 179
  14. 2020Geometric Dataset Distances via Optimal Transportcited by 157
  15. 2021Well-tuned Simple Nets Excel on Tabular Datasetscited by 151
  16. 2019MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-IIIcited by 121
  17. 2020The Effect of Natural Distribution Shift on Question Answering Modelscited by 104
  18. 2021Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Taskscited by 100
  19. 2022DataPerf: Benchmarks for Data-Centric AI Developmentcited by 89
  20. 2019An Open Source AutoML Benchmarkcited by 85
  21. 2014Domain-Adversarial Neural Networkscited by 78
  22. 2020Coping with Label Shift via Distributionally Robust Optimisationcited by 69
  23. 2021It's COMPASlicated: The Messy Relationship between RAI Datasets and Algorithmic Fairness Benchmarkscited by 61
  24. 2021DORO: Distributional and Outlier Robust Optimizationcited by 54
  25. 2019ProPublica's COMPAS Data Revisitedcited by 24
  26. 2022Exploring The Landscape of Distributional Robustness for Question Answering Modelscited by 23
  27. 2022Shifts 2.0: Extending The Dataset of Real Distributional Shiftscited by 22
  28. 2022Subgroup Robustness Grows On Trees: An Empirical Baseline Investigationcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 25; 2026: 18.

64 papers.

Sort:

  1. 2024Why Tabular Foundation Models Should Be a Research Prioritycited by 103
  2. 2024Large Scale Transfer Learning for Tabular Data via Language Modelingcited by 73
  3. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  4. 2024TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarkscited by 29
  5. 2024Drift-Resilient TabPFN: In-Context Learning Temporal Distribution Shifts on Tabular Datacited by 25
  6. 2024BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Modelcited by 21
  7. 2024Text Serialization and Their Relationship with the Conventional Paradigms of Tabular Machine Learningcited by 17
  8. 2024The Landscape of Causal Discovery Data: Grounding Causal Discovery in Real-World Applicationscited by 17
  9. 2024A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Datacited by 15
  10. 2024TabularBench: Benchmarking Adversarial Robustness for Tabular Deep Learning in Real-world Use-casescited by 10
  11. 2024Estimating Model Performance Under Covariate Shift Without Labelscited by 9
  12. 2024Confidence-based Estimators for Predictive Performance in Model Monitoringcited by 8
38 more
  1. 2024ABCFair: an Adaptable Benchmark approach for Comparing Fairness Methodscited by 8
  2. 2024Fully Test-time Adaptation for Tabular Datacited by 8
  3. 2025Towards Fair In-Context Learning with Tabular Foundation Modelscited by 8
  4. 2024Causality for Tabular Data Synthesis: A High-Order Structure Causal Benchmark Frameworkcited by 7
  5. 2024AdapTable: Test-Time Adaptation for Tabular Data via Shift-Aware Uncertainty Calibrator and Label Distribution Handlercited by 6
  6. 2025CLIMB: Class-imbalanced Learning Benchmark on Tabular Datacited by 6
  7. 2025SIMSHIFT: A Benchmark for Adapting Neural Surrogates to Distribution Shiftscited by 6
  8. 2025Keeping Medical AI Healthy and Trustworthy: A Review of Detection and Correction Methods for System Degradationcited by 6
  9. 2024Quantifying Prediction Consistency Under Fine-Tuning Multiplicity in Tabular LLMscited by 5
  10. 2025When Shift Happens - Confounding Is to Blamecited by 5
  11. 2025DRO: A Python Library for Distributionally Robust Optimization in Machine Learningcited by 5
  12. 2026End-to-End Compression for Tabular Foundation Modelscited by 5
  13. 2024Multi-Source and Test-Time Domain Adaptation on Multivariate Signals using Spatio-Temporal Monge Alignmentcited by 4
  14. 2025Understanding the Limits of Deep Tabular Methods with Temporal Shiftcited by 4
  15. 2025Realistic Evaluation of TabPFN v2 in Open Environmentscited by 3
  16. 2026Mitigating Label Shift in Tabular In-Context Learning via Test-Time Posterior Adjustmentcited by 3
  17. 2024SKADA-Bench: Benchmarking Unsupervised Domain Adaptation Methods with Realistic Validation On Diverse Modalitiescited by 2
  18. 2025Synthetic Tabular Data Detection In the Wildcited by 2
  19. 2025Learning Decision Trees as Amortized Structure Inferencecited by 2
  20. 2025Performance Estimation in Binary Classification Using Calibrated Confidencecited by 2
  21. 2025Wasserstein Distances Made Explainable: Insights Into Dataset Shifts and Transport Phenomenacited by 2
  22. 2025An Investigation of Visual Foundation Models Robustnesscited by 2
  23. 2025Predicting Language Models' Success at Zero-Shot Probabilistic Predictioncited by 2
  24. 2026LakeMLB: Data Lake Machine Learning Benchmarkcited by 2
  25. 2024Reliable OOD Virtual Screening with Extrapolatory Pseudo-Label Matchingcited by 1
  26. 2025Leveraging Machine Learning Models to Predict the Outcome of Digital Medical Triage Interviewscited by 1
  27. 2025ADAPT: A Pseudo-labeling Approach to Combat Concept Drift in Malware Detectioncited by 1
  28. 2026GHGbench: A Unified Multi-Entity, Multi-Task Benchmark for Carbon Emission Predictioncited by 1
  29. 2024Table2Image: Lightweight Tabular Learning with Generated Proxy Representations and Reliability Diagnosticscited by 0
  30. 2024Cross-table Synthetic Tabular Data Detectioncited by 0
  31. 2025Robust Predictive Modeling Under Unseen Data Distribution Shifts: A Methodological Commentarycited by 0
  32. 2025Prediction Models That Learn to Avoid Missing Valuescited by 0
  33. 2025MultiTab: A Comprehensive Benchmark Suite for Multi-Dimensional Evaluation in Tabular Domainscited by 0
  34. 2025Explaining Concept Shift with Interpretable Feature Attributioncited by 0
  35. 2025ICE-ID: A Novel Historical Census Dataset for Longitudinal Identity Resolutioncited by 0
  36. 2025Technical note on Fisher Information for Robust Federated Cross-Validationcited by 0
  37. 2025Aeolus: A Multi-structural Flight Delay Datasetcited by 0
  38. 2025Feature-aware Modulation for Learning from Temporal Tabular Datacited by 0

50 most cited of 64 All 64 (JSON)