Data Quality Awareness: A Journey from Traditional Data Management to Data Science Systems
Not on the map. Why?
Abstract (arXiv)
Artificial intelligence (AI) has transformed various fields, significantly impacting our daily lives. A major factor in AI success is high-quality data. In this paper, we present a comprehensive review of the evolution of data quality (DQ) awareness from traditional data management systems to modern data-driven AI systems, which are integral to data science. We synthesize the existing literature, highlighting the quality challenges and techniques that have evolved from traditional data management to data science including big data and ML fields. As data science systems support a wide range of activities, our focus in this paper lies specifically in the analytics aspect driven by machine learning. We use the cause-effect connection between the quality challenges of ML and those of big data to allow a more thorough understanding of emerging DQ challenges and the related quality awareness techniques in data science systems. To the best of our knowledge, our paper is the first to provide a review of DQ awareness spanning traditional and emergent data science systems. We hope that readers will find this journey through the evolution of data quality awareness insightful and valuable.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 146 references that are in the map. First posted on arXiv 5 Nov 2024.
Builds on
29 of 146 references.
References · 29
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
- 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2015Domain-Adversarial Training of Neural Networkscited by 3,023
- 2022Survey of Hallucination in Natural Language Generationcited by 2,056
- 2016Equality of Opportunity in Supervised Learningcited by 2,000
- 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,543
- 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
- 2014Certifying and removing disparate impactcited by 715
- 2018Mitigating Unwanted Biases with Adversarial Learningcited by 663
- 2017Snorkel: Rapid Training Data Creation with Weak Supervisioncited by 370
17 more
- 2018GAIN: Missing Data Imputation using Generative Adversarial Netscited by 349
- 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 266
- 2016Revisiting Classifier Two-Sample Testscited by 258
- 2020Learning Deep Kernels for Non-Parametric Two-Sample Testscited by 129
- 2022HyperImpute: Generalized Iterative Imputation with Automatic Model Selectioncited by 81
- 2017HoloClean: Holistic Data Repairs with Probabilistic Inferencecited by 79
- 2021Socially Responsible AI Algorithms: Issues, Purposes, and Challengescited by 56
- 2015Fast Two-Sample Testing with Analytic Representations of Probability Measurescited by 53
- 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
- 2020Towards CRISP-ML(Q): A Machine Learning Process Model with Quality Assurance Methodologycited by 41
- 2022The Effects of Data Quality on Machine Learning Performance on Tabular Datacited by 19
- 2021Data Quality Toolkit: Automatic assessment of data quality and remediation for machine learning datasetscited by 18
- 2020Towards Guidelines for Assessing Qualities of Machine Learning Systemscited by 16
- 2022Use of Context in Data Quality Management: a Systematic Literature Reviewcited by 3
- 2016Operationalised product quality models and assessment: The Quamoco approachcited by 2
- 2022Modeling Quality and Machine Learning Pipelines through Extended Feature Modelscited by 2
- 2024Data Quality Assessment: Challenges and Opportunitiescited by 2
Built on by
Not cited yet.