# Data-IQ: Characterizing subgroups with heterogeneous outcomes in tabular data

arXiv: https://arxiv.org/abs/2210.13043 · HTML: https://mlsanity.com/paper/2210.13043/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 58 references that are in the map.
- First posted on arXiv 24 Oct 2022.

## Builds on (most cited first, 30 of 35)

- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Modeling Tabular data using Conditional GAN](https://mlsanity.com/paper/1907.00503/) (2019)
- [Data Shapley: Equitable Valuation of Data for Machine Learning](https://mlsanity.com/paper/1904.02868/) (2019)
- [Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks](https://mlsanity.com/paper/1901.08584/) (2019)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [Just Train Twice: Improving Group Robustness without Training Group Information](https://mlsanity.com/paper/2107.09044/) (2021)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics](https://mlsanity.com/paper/2009.10795/) (2020)
- [Fairness Without Demographics in Repeated Loss Minimization](https://mlsanity.com/paper/1806.08010/) (2018)
- [Real-valued (Medical) Time Series Generation with Recurrent Conditional GANs](https://mlsanity.com/paper/1706.02633/) (2017)
- [Identifying Mislabeled Data using the Area Under the Margin Ranking](https://mlsanity.com/paper/2001.10528/) (2020)
- [No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems](https://mlsanity.com/paper/2011.12945/) (2020)
- [How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models](https://mlsanity.com/paper/2102.08921/) (2021)
- [Evaluating Bayesian Deep Learning Methods for Semantic Segmentation](https://mlsanity.com/paper/1811.12709/) (2018)
- [Improving model calibration with accuracy versus uncertainty optimization](https://mlsanity.com/paper/2012.07923/) (2020)
- [Ethical Machine Learning in Health Care](https://mlsanity.com/paper/2009.10576/) (2020)
- [How Complex is your classification problem? A survey on measuring classification complexity](https://mlsanity.com/paper/1808.03591/) (2018)
- [Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors](https://mlsanity.com/paper/1806.05975/) (2018)
- [A Data Quality-Driven View of MLOps](https://mlsanity.com/paper/2102.07750/) (2021)
- [What can Data-Centric AI Learn from Data and ML Engineering?](https://mlsanity.com/paper/2112.06439/) (2021)
- [Data Quality Toolkit: Automatic assessment of data quality and remediation for machine learning datasets](https://mlsanity.com/paper/2108.05935/) (2021)
- [SYNC: A Copula based Framework for Generating Synthetic Data from Aggregated Sources](https://mlsanity.com/paper/2009.09471/) (2020)
- [Towards Equity and Algorithmic Fairness in Student Grade Prediction](https://mlsanity.com/paper/2105.06604/) (2021)
- [Data-SUITE: Data-centric identification of in-distribution incongruous examples](https://mlsanity.com/paper/2202.08836/) (2022)
- [Towards calibrated and scalable uncertainty representations for neural networks](https://mlsanity.com/paper/1911.00104/) (2019)

## Built on by (most cited first, 22 of 22)

- [Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimes](https://mlsanity.com/paper/2312.12112/) (2023)
- [Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Data](https://mlsanity.com/paper/2310.16524/) (2023)
- [Reimagining Synthetic Tabular Data Generation through Data-Centric AI: A Comprehensive Benchmark](https://mlsanity.com/paper/2310.16981/) (2023)
- [DC-Check: A Data-Centric AI checklist to guide the development of reliable machine learning systems](https://mlsanity.com/paper/2211.05764/) (2022)
- [DMLR: Data-centric Machine Learning Research -- Past, Present and Future](https://mlsanity.com/paper/2311.13028/) (2023)
- [Dissecting Sample Hardness: A Fine-Grained Analysis of Hardness Characterization Methods for Data-Centric AI](https://mlsanity.com/paper/2403.04551/) (2024)
- [TRIAGE: Characterizing and auditing training data for improved regression](https://mlsanity.com/paper/2310.18970/) (2023)
- [Selective Learning: Towards Robust Calibration with Dynamic Regularization](https://mlsanity.com/paper/2402.08384/) (2024)
- [When is Off-Policy Evaluation (Reward Modeling) Useful in Contextual Bandits? A Data-Centric Perspective](https://mlsanity.com/paper/2311.14110/) (2023)
- [Targeted synthetic data generation for tabular data via hardness characterization](https://mlsanity.com/paper/2410.00759/) (2024)
- [You can't handle the (dirty) truth: Data-centric insights improve pseudo-labeling](https://mlsanity.com/paper/2406.13733/) (2024)
- [CliMB: An AI-enabled Partner for Clinical Predictive Modeling](https://mlsanity.com/paper/2410.03736/) (2024)
- [Mislabeled examples detection viewed as probing machine learning models: concepts, survey and extensive benchmark](https://mlsanity.com/paper/2410.15772/) (2024)
- [U-PASS: an Uncertainty-guided deep learning Pipeline for Automated Sleep Staging](https://mlsanity.com/paper/2306.04663/) (2023)
- [VTruST: Controllable value function based subset selection for Data-Centric Trustworthy AI](https://mlsanity.com/paper/2403.05174/) (2024)
- [Data Heterogeneity Modeling for Trustworthy Machine Learning](https://mlsanity.com/paper/2506.00969/) (2025)
- [Pseudo-D: Informing Multi-View Uncertainty Estimation with Calibrated Neural Training Dynamics](https://mlsanity.com/paper/2509.11800/) (2025)
- [Draw a Portrait of Your Graph Data: An Instance-Level Profiling Framework for Graph-Structured Data](https://mlsanity.com/paper/2509.12094/) (2025)
- [LiLAW: Lightweight Learnable Adaptive Weighting to Learn Sample Difficulty & Improve Noisy Training](https://mlsanity.com/paper/2509.20786/) (2025)
- [Filtering instances and rejecting predictions to obtain reliable models in healthcare](https://mlsanity.com/paper/2510.24368/) (2025)
- [What's the next frontier for Data-centric AI? Data Savvy Agents](https://mlsanity.com/paper/2511.01015/) (2025)
- [Not All Confusion Is Equal: A Source-Aware Uncertainty Diagnosis for Fine-Grained Aircraft Detection](https://mlsanity.com/paper/2609.29959/) (2026)

## Abstract (arXiv)

High model performance, on average, can hide that models may systematically underperform on subgroups of the data. We consider the tabular setting, which surfaces the unique issue of outcome heterogeneity - this is prevalent in areas such as healthcare, where patients with similar features can have different outcomes, thus making reliable predictions challenging. To tackle this, we propose Data-IQ, a framework to systematically stratify examples into subgroups with respect to their outcomes. We do this by analyzing the behavior of individual examples during training, based on their predictive confidence and, importantly, the aleatoric (data) uncertainty. Capturing the aleatoric uncertainty permits a principled characterization and then subsequent stratification of data examples into three distinct subgroups (Easy, Ambiguous, Hard). We experimentally demonstrate the benefits of Data-IQ on four real-world medical datasets. We show that Data-IQ's characterization of examples is most robust to variation across similarly performant (yet different) models, compared to baselines. Since Data-IQ can be used with any ML model (including neural networks, gradient boosting etc.), this property ensures consistency of data characterization, while allowing flexible model selection. Taking this a step further, we demonstrate that the subgroups enable us to construct new approaches to both feature acquisition and dataset selection. Furthermore, we highlight how the subgroups can inform reliable model usage, noting the significant impact of the Ambiguous subgroup on model generalization.