mlsanityData to 7 Oct 2026

DC-Check: A Data-Centric AI checklist to guide the development of reliable machine learning systems

arXiv 2211.05764 ↗First posted 9 Nov 2022cs.LG cs.AI cs.CYcited by 14

Not on the map. Why?

Abstract (arXiv)

While there have been a number of remarkable breakthroughs in machine learning (ML), much of the focus has been placed on model development. However, to truly realize the potential of machine learning in real-world settings, additional aspects must be considered across the ML pipeline. Data-centric AI is emerging as a unifying paradigm that could enable such reliable end-to-end pipelines. However, this remains a nascent area with no standardized framework to guide practitioners to the necessary data-centric considerations or to communicate the design of data-centric driven ML systems. To address this gap, we propose DC-Check, an actionable checklist-style framework to elicit data-centric considerations at different stages of the ML pipeline: Data, Training, Testing, and Deployment. This data-centric lens on development aims to promote thoughtfulness and transparency prior to system development. Additionally, we highlight specific data-centric AI challenges and research opportunities. DC-Check is aimed at both practitioners and researchers to guide day-to-day development. As such, to easily engage with and use DC-Check and associated resources, we provide a DC-Check companion website (https://www.vanderschaar-lab.com/dc-check/). The website will also serve as an updated resource as methods and tooling evolve over time.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 86 of its 177 references that are in the map. First posted on arXiv 9 Nov 2022.

Builds on

86 of 177 references.

References · 86

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  5. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  6. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  7. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  8. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  9. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  10. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  11. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  12. 2016Equality of Opportunity in Supervised Learningcited by 2,000
38 more
  1. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,906
  2. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  3. 2018Datasheets for Datasetscited by 1,371
  4. 2015Learning Transferable Features with Deep Adaptation Networkscited by 1,323
  5. 2020Energy-based Out-of-distribution Detectioncited by 1,296
  6. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  7. 2019A Comprehensive Survey on Transfer Learningcited by 1,154
  8. 2018Model Cards for Model Reportingcited by 1,068
  9. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,053
  10. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,037
  11. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
  12. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  13. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 903
  14. 2019Modeling Tabular data using Conditional GANcited by 695
  15. 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 652
  16. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 617
  17. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 580
  18. 2021Tabular Data: Deep Learning is Not All You Needcited by 530
  19. 2019Likelihood Ratios for Out-of-Distribution Detectioncited by 451
  20. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
  21. 2019Confident Learning: Estimating Uncertainty in Dataset Labelscited by 411
  22. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 388
  23. 2018GAIN: Missing Data Imputation using Generative Adversarial Netscited by 349
  24. 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 329
  25. 2015Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networkscited by 295
  26. 2017Data Augmentation Generative Adversarial Networkscited by 278
  27. 2020Identifying Mislabeled Data using the Area Under the Margin Rankingcited by 259
  28. 2016Revisiting Classifier Two-Sample Testscited by 258
  29. 2020Normalized Loss Functions for Deep Learning with Noisy Labelscited by 257
  30. 2016Data Programming: Creating Large Training Sets, Quicklycited by 246
  31. 2018Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shiftcited by 240
  32. 2018Techniques for Interpretable Machine Learningcited by 237
  33. 2019Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imagingcited by 215
  34. 2018A comparative study of fairness-enhancing interventions in machine learningcited by 208
  35. 2020No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problemscited by 208
  36. 2020Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scanscited by 158
  37. 2022Domino: Discovering Systematic Errors with Cross-Modal Embeddingscited by 155
  38. 2016Correlation Alignment for Unsupervised Domain Adaptationcited by 146

50 most cited of 86 All 86 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 6; 2025: 5.

14 papers.

Sort:

By area:

  1. 2023DMLR: Data-centric Machine Learning Research -- Past, Present and Futurecited by 11
  2. 2024Matchmaker: Self-Improving Large Language Model Programs for Schema Matchingcited by 9
  3. 2024Context-Aware Testing: A New Paradigm for Model Testing with Large Language Modelscited by 8
  4. 2022Data Leakage and Evaluation Issues in Micro-Expression Analysiscited by 7
  5. 2025OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Modelscited by 5
  6. 2023When is Off-Policy Evaluation (Reward Modeling) Useful in Contextual Bandits? A Data-Centric Perspectivecited by 3
  7. 2024You can't handle the (dirty) truth: Data-centric insights improve pseudo-labelingcited by 3
  8. 2024Data-Centric Improvements for Enhancing Multi-Modal Understanding in Spoken Conversation Modelingcited by 3
  9. 2025Towards deployment-centric multimodal AI beyond vision and languagecited by 2
  10. 2025End-to-End Data Quality-Driven Framework for Machine Learning in Production Environmentcited by 1
  11. 2024AI Competitions and Benchmarks: Dataset Developmentcited by 0
  12. 2024Unlocking Historical Clinical Trial Data with ALIGN: A Compositional Large Language Model System for Medical Codingcited by 0
2 more
  1. 2025What's the next frontier for Data-centric AI? Data Savvy Agentscited by 0
  2. 2025Evaluation of AI Ethics Tools in Language Models: A Developers' Perspective Case Studycited by 0