# Turning the Tables: Biased, Imbalanced, Dynamic Tabular Datasets for ML Evaluation

arXiv: https://arxiv.org/abs/2211.13358 · HTML: https://mlsanity.com/paper/2211.13358/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 55 references that are in the map.
- First posted on arXiv 24 Nov 2022.

## Builds on (most cited first, 28 of 28)

- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Modeling Tabular data using Conditional GAN](https://mlsanity.com/paper/1907.00503/) (2019)
- [Retiring Adult: New Datasets for Fair Machine Learning](https://mlsanity.com/paper/2108.04884/) (2021)
- [Algorithmic decision making and the cost of fairness](https://mlsanity.com/paper/1701.08230/) (2017)
- [Performative Prediction](https://mlsanity.com/paper/2002.06673/) (2020)
- [Data and its (dis)contents: A survey of dataset development and use in machine learning research](https://mlsanity.com/paper/2012.05345/) (2020)
- [Differentially Private Generative Adversarial Network](https://mlsanity.com/paper/1802.06739/) (2018)
- [A comparative study of fairness-enhancing interventions in machine learning](https://mlsanity.com/paper/1802.04422/) (2018)
- [Fairness in Machine Learning](https://mlsanity.com/paper/2012.15816/) (2020)
- [A survey on datasets for fairness-aware machine learning](https://mlsanity.com/paper/2110.00530/) (2021)
- [CTAB-GAN: Effective Table Data Synthesizing](https://mlsanity.com/paper/2102.08369/) (2021)
- [Pros and Cons of GAN Evaluation Measures: New Developments](https://mlsanity.com/paper/2103.09396/) (2021)
- [Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research](https://mlsanity.com/paper/2112.01716/) (2021)
- [Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks](https://mlsanity.com/paper/2107.07455/) (2021)
- [Benchmarking Differentially Private Synthetic Data Generation Algorithms](https://mlsanity.com/paper/2112.09238/) (2021)
- [It's COMPASlicated: The Messy Relationship between RAI Datasets and Algorithmic Fairness Benchmarks](https://mlsanity.com/paper/2106.05498/) (2021)
- [Algorithmic Fairness Datasets: the Story so Far](https://mlsanity.com/paper/2202.01711/) (2022)
- [Differentially Private Synthetic Data: Applied Evaluations and Enhancements](https://mlsanity.com/paper/2011.05537/) (2020)
- [The AI Index 2021 Annual Report](https://mlsanity.com/paper/2103.06312/) (2021)
- [Differentially Private Data Generative Models](https://mlsanity.com/paper/1812.02274/) (2018)
- [Value-laden Disciplinary Shifts in Machine Learning](https://mlsanity.com/paper/1912.01172/) (2019)
- [ProPublica's COMPAS Data Revisited](https://mlsanity.com/paper/1906.04711/) (2019)
- [A Framework for Deprecating Datasets: Standardizing Documentation, Identification, and Communication](https://mlsanity.com/paper/2111.04424/) (2021)
- [An Empirical Comparison of Bias Reduction Methods on Real-World Problems in High-Stakes Policy Settings](https://mlsanity.com/paper/2105.06442/) (2021)
- [Understanding Unfairness in Fraud Detection through Model and Data Bias Interactions](https://mlsanity.com/paper/2207.06273/) (2022)
- [Utilizing Noise Addition for Data Privacy, an Overview](https://mlsanity.com/paper/1309.3958/) (2013)
- [Prisoners of Their Own Devices: How Models Induce Data Bias in Performative Prediction](https://mlsanity.com/paper/2206.13183/) (2022)

## Built on by (most cited first, 30 of 40)

- [Representation Learning for Tabular Data: A Comprehensive Survey](https://mlsanity.com/paper/2504.16109/) (2025)
- [Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Data](https://mlsanity.com/paper/2310.16524/) (2023)
- [A Survey on Evaluation of Out-of-Distribution Generalization](https://mlsanity.com/paper/2403.01874/) (2024)
- [Transparency and Privacy: The Role of Explainable AI and Federated Learning in Financial Fraud Detection](https://mlsanity.com/paper/2312.13334/) (2023)
- [Cost-Sensitive Learning to Defer to Multiple Experts with Workload Constraints](https://mlsanity.com/paper/2403.06906/) (2024)
- [Balanced Mixed-Type Tabular Data Synthesis with Diffusion Models](https://mlsanity.com/paper/2404.08254/) (2024)
- [Causality for Tabular Data Synthesis: A High-Order Structure Causal Benchmark Framework](https://mlsanity.com/paper/2406.08311/) (2024)
- [CLIMB: Class-imbalanced Learning Benchmark on Tabular Data](https://mlsanity.com/paper/2505.17451/) (2025)
- [FairJob: A Real-World Dataset for Fairness in Online Systems](https://mlsanity.com/paper/2407.03059/) (2024)
- [Fairness-Aware Data Valuation for Supervised Learning](https://mlsanity.com/paper/2303.16963/) (2023)
- [Aequitas Flow: Streamlining Fair ML Experimentation](https://mlsanity.com/paper/2405.05809/) (2024)
- [Exposing the Illusion of Fairness: Auditing Vulnerabilities to Distributional Manipulation Attacks](https://mlsanity.com/paper/2507.20708/) (2025)
- [FAMOSE: A ReAct Approach to Automated Feature Discovery](https://mlsanity.com/paper/2602.17641/) (2026)
- [FiFAR: A Fraud Detection Dataset for Learning to Defer](https://mlsanity.com/paper/2312.13218/) (2023)
- [Is it Still Fair? A Comparative Evaluation of Fairness Algorithms through the Lens of Covariate Drift](https://mlsanity.com/paper/2409.12428/) (2024)
- [Kolmogorov Arnold Networks in Fraud Detection: Bridging the Gap Between Theory and Practice](https://mlsanity.com/paper/2408.10263/) (2024)
- [Transferable Adversarial Robustness for Categorical Data via Universal Robust Embeddings](https://mlsanity.com/paper/2306.04064/) (2023)
- [Decoupling Decision-Making in Fraud Prevention through Classifier Calibration for Business Logic Action](https://mlsanity.com/paper/2401.05240/) (2024)
- [A Note on Bias to Complete](https://mlsanity.com/paper/2402.11710/) (2024)
- [RIFF: Inducing Rules for Fraud Detection from Decision Trees](https://mlsanity.com/paper/2408.12989/) (2024)
- [Fair-OBNC: Correcting Label Noise for Fairer Datasets](https://mlsanity.com/paper/2410.06214/) (2024)
- [PnPXAI: A Universal XAI Framework Providing Automatic Explanations Across Diverse Modalities and Models](https://mlsanity.com/paper/2505.10515/) (2025)
- [Fair for a few: Improving Fairness in Doubly Imbalanced Datasets](https://mlsanity.com/paper/2506.14306/) (2025)
- [Evaluating Transfer Learning Methods on Real-World Data Streams: A Case Study in Financial Fraud Detection](https://mlsanity.com/paper/2508.02702/) (2025)
- [On Ranking-based Tests of Independence](https://mlsanity.com/paper/2403.07464/) (2024)
- ["Show Me What's Wrong!": Combining Charts and Text to Guide Data Analysis](https://mlsanity.com/paper/2410.00727/) (2024)
- [Undermining Image and Text Classification Algorithms Using Adversarial Attacks](https://mlsanity.com/paper/2411.03348/) (2024)
- [A Critical Review of Predominant Bias in Neural Networks](https://mlsanity.com/paper/2502.11031/) (2025)
- [Data Heterogeneity Modeling for Trustworthy Machine Learning](https://mlsanity.com/paper/2506.00969/) (2025)
- [GFLC: Graph-based Fairness-aware Label Correction for Fair Classification](https://mlsanity.com/paper/2506.15620/) (2025)

## Abstract (arXiv)

Evaluating new techniques on realistic datasets plays a crucial role in the development of ML research and its broader adoption by practitioners. In recent years, there has been a significant increase of publicly available unstructured data resources for computer vision and NLP tasks. However, tabular data -- which is prevalent in many high-stakes domains -- has been lagging behind. To bridge this gap, we present Bank Account Fraud (BAF), the first publicly available privacy-preserving, large-scale, realistic suite of tabular datasets. The suite was generated by applying state-of-the-art tabular data generation techniques on an anonymized,real-world bank account opening fraud detection dataset. This setting carries a set of challenges that are commonplace in real-world applications, including temporal dynamics and significant class imbalance. Additionally, to allow practitioners to stress test both performance and fairness of ML methods, each dataset variant of BAF contains specific types of data bias. With this resource, we aim to provide the research community with a more realistic, complete, and robust test bed to evaluate novel and existing methods.