# Still More Shades of Null: An Evaluation Suite for Responsible Missing Value Imputation

arXiv: https://arxiv.org/abs/2409.07510 · HTML: https://mlsanity.com/paper/2409.07510/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 104 references that are in the map.
- First posted on arXiv 11 Sep 2024.

## Builds on (most cited first, 30 of 31)

- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [Certifying and removing disparate impact](https://mlsanity.com/paper/1412.3756/) (2014)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [Retiring Adult: New Datasets for Fair Machine Learning](https://mlsanity.com/paper/2108.04884/) (2021)
- [GAIN: Missing Data Imputation using Generative Adversarial Nets](https://mlsanity.com/paper/1806.02920/) (2018)
- [Auto-Keras: An Efficient Neural Architecture Search System](https://mlsanity.com/paper/1806.10282/) (2018)
- [A survey on datasets for fairness-aware machine learning](https://mlsanity.com/paper/2110.00530/) (2021)
- [Handling Incomplete Heterogeneous Data using VAEs](https://mlsanity.com/paper/1807.03653/) (2018)
- [Survey of multifidelity methods in uncertainty propagation, inference, and optimization](https://mlsanity.com/paper/1806.10761/) (2018)
- [not-MIWAE: Deep Generative Modelling with Missing not at Random Data](https://mlsanity.com/paper/2006.12871/) (2020)
- [What's a good imputation to predict with missing values?](https://mlsanity.com/paper/2106.00311/) (2021)
- [Transformed Distribution Matching for Missing Value Imputation](https://mlsanity.com/paper/2302.10363/) (2023)
- [Identifiable Generative Models for Missing Not at Random Data Imputation](https://mlsanity.com/paper/2110.14708/) (2021)
- [A Validity Perspective on Evaluating the Justified Use of Data-driven Decision-making Algorithms](https://mlsanity.com/paper/2206.14983/) (2022)
- [Handling missing values in healthcare data: A systematic review of deep learning-based imputation techniques](https://mlsanity.com/paper/2210.08258/) (2022)
- [Fairness without Imputation: A Decision Tree Approach for Fair Prediction with Missing Values](https://mlsanity.com/paper/2109.10431/) (2021)
- [Assessing Fairness in the Presence of Missing Data](https://mlsanity.com/paper/2112.04899/) (2021)
- [Learning from data with structured missingness](https://mlsanity.com/paper/2304.01429/) (2023)
- [BoostClean: Automated Error Detection and Repair for Machine Learning](https://mlsanity.com/paper/1711.01299/) (2017)
- [Classification of datasets with imputed missing values: does imputation quality matter?](https://mlsanity.com/paper/2206.08478/) (2022)
- [Nearest Neighbor Classifiers over Incomplete Information: From Certain Answers to Certain Predictions](https://mlsanity.com/paper/2005.05117/) (2020)
- [Fairness and Missing Values](https://mlsanity.com/paper/1905.12728/) (2019)
- [An External Stability Audit Framework to Test the Validity of Personality Prediction in AI Hiring](https://mlsanity.com/paper/2201.09151/) (2022)
- [Towards Substantive Conceptions of Algorithmic Fairness: Normative Guidance from Equal Opportunity Doctrines](https://mlsanity.com/paper/2207.02912/) (2022)
- [REIN: A Comprehensive Benchmark Framework for Data Cleaning Methods in ML Pipelines](https://mlsanity.com/paper/2302.04702/) (2023)
- [Fairness in Missing Data Imputation](https://mlsanity.com/paper/2110.12002/) (2021)
- [Adapting Fairness Interventions to Missing Values](https://mlsanity.com/paper/2305.19429/) (2023)
- [Arbitrariness Lies Beyond the Fairness-Accuracy Frontier](https://mlsanity.com/paper/2306.09425/) (2023)
- [A Survey on Uncertainty Toolkits for Deep Learning](https://mlsanity.com/paper/2205.01040/) (2022)

## Built on by (most cited first, 1 of 1)

- [VirnyFlow: Optimizing ML Pipelines for Accuracy, Fairness, and Stability at Scale](https://mlsanity.com/paper/2506.01584/) (2025)

## Abstract (arXiv)

Data missingness is a practical challenge of sustained interest to the scientific community. In this paper, we present Shades-of-Null, an evaluation suite for responsible missing value imputation. Our work is novel in two ways (i) we model realistic and socially-salient missingness scenarios that go beyond Rubin's classic Missing Completely at Random (MCAR), Missing At Random (MAR) and Missing Not At Random (MNAR) settings, to include multi-mechanism missingness (when different missingness patterns co-exist in the data) and missingness shift (when the missingness mechanism changes between training and test) (ii) we evaluate imputers holistically, based on imputation quality and imputation fairness, as well as on the predictive performance, fairness and stability of the models that are trained and tested on the data post-imputation. We use Shades-of-Null to conduct a large-scale empirical study involving 29,736 experimental pipelines, and find that while there is no single best-performing imputation approach for all missingness types, interesting trade-offs arise between predictive performance, fairness and stability, based on the combination of missingness scenario, imputer choice, and the architecture of the predictive model. We make Shades-of-Null publicly available, to enable researchers to rigorously evaluate missing value imputation methods on a wide range of metrics in plausible and socially meaningful scenarios.