# Collect, Measure, Repeat: Reliability Factors for Responsible AI Data Collection arXiv: https://arxiv.org/abs/2308.12885 ยท HTML: https://mlsanity.com/paper/2308.12885/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Builds on 28 of its 108 references that are in the map. - First posted on arXiv 22 Aug 2023. ## Builds on (most cited first, 28 of 28) - [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019) - [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016) - [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018) - [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018) - [Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints](https://mlsanity.com/paper/1707.09457/) (2017) - [Data and its (dis)contents: A survey of dataset development and use in machine learning research](https://mlsanity.com/paper/2012.05345/) (2020) - [Women also Snowboard: Overcoming Bias in Captioning Models](https://mlsanity.com/paper/1803.09797/) (2018) - [Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI](https://mlsanity.com/paper/2204.01075/) (2022) - [Aequitas: A Bias and Fairness Audit Toolkit](https://mlsanity.com/paper/1811.05577/) (2018) - [Toward a Perspectivist Turn in Ground Truthing for Predictive Computing](https://mlsanity.com/paper/2109.04270/) (2021) - [Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels](https://mlsanity.com/paper/2101.05022/) (2021) - [Garbage In, Garbage Out? Do Machine Learning Application Papers in Social Computing Report Where Human-Labeled Training Data Comes From?](https://mlsanity.com/paper/1912.08320/) (2019) - [CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation](https://mlsanity.com/paper/2206.08931/) (2022) - [Twitter as a Lifeline: Human-annotated Twitter Corpora for NLP of Crisis-related Messages](https://mlsanity.com/paper/1605.05894/) (2016) - [Cross-replication Reliability -- An Empirical Approach to Interpreting Inter-rater Reliability](https://mlsanity.com/paper/2106.07393/) (2021) - [CrowdTruth 2.0: Quality Metrics for Crowdsourcing with Disagreement](https://mlsanity.com/paper/1808.06080/) (2018) - [Sentiment Analysis in the News](https://mlsanity.com/paper/1309.6202/) (2013) - [Measuring Annotator Agreement Generally across Complex Structured, Multi-object, and Free-text Annotation Tasks](https://mlsanity.com/paper/2212.09503/) (2022) - [Incentivizing High Quality Crowdwork](https://mlsanity.com/paper/1503.05897/) (2015) - [Metrology for AI: From Benchmarks to Instruments](https://mlsanity.com/paper/1911.01875/) (2019) - [Studying the Effects of Cognitive Biases in Evaluation of Conversational Agents](https://mlsanity.com/paper/2002.07927/) (2020) - [Quantified Reproducibility Assessment of NLP Results](https://mlsanity.com/paper/2204.05961/) (2022) - [Evaluation of Automatic Video Captioning Using Direct Assessment](https://mlsanity.com/paper/1710.10586/) (2017) - [Empirical Methodology for Crowdsourcing Ground Truth](https://mlsanity.com/paper/1809.08888/) (2018) - [Much Ado About Time: Exhaustive Annotation of Temporal Data](https://mlsanity.com/paper/1607.07429/) (2016) - [On the state of reporting in crowdsourcing experiments and a checklist to aid current practices](https://mlsanity.com/paper/2107.13519/) (2021) - [Crowd-Labeling Fashion Reviews with Quality Control](https://mlsanity.com/paper/1805.09648/) (2018) - [Eliciting User Preferences for Personalized Explanations for Video Summaries](https://mlsanity.com/paper/2005.00465/) (2020) ## Built on by (most cited first, 5 of 5) - [Large language models can accurately predict searcher preferences](https://mlsanity.com/paper/2309.10621/) (2023) - [Towards Estimating Personal Values in Song Lyrics](https://mlsanity.com/paper/2408.12694/) (2024) - [Bridging the Data Gap in AI Reliability Research and Establishing DR-AIR, a Comprehensive Data Repository for AI Reliability](https://mlsanity.com/paper/2502.12386/) (2025) - [Key Considerations for Domain Expert Involvement in LLM Design and Evaluation: An Ethnographic Study](https://mlsanity.com/paper/2602.14357/) (2026) - [Toward Trustworthy Evaluation of Sustainability Rating Methodologies: A Human-AI Collaborative Framework for Benchmark Dataset Construction](https://mlsanity.com/paper/2602.17106/) (2026) ## Abstract (arXiv) The rapid entry of machine learning approaches in our daily activities and high-stakes domains demands transparency and scrutiny of their fairness and reliability. To help gauge machine learning models' robustness, research typically focuses on the massive datasets used for their deployment, e.g., creating and maintaining documentation for understanding their origin, process of development, and ethical considerations. However, data collection for AI is still typically a one-off practice, and oftentimes datasets collected for a certain purpose or application are reused for a different problem. Additionally, dataset annotations may not be representative over time, contain ambiguous or erroneous annotations, or be unable to generalize across issues or domains. Recent research has shown these practices might lead to unfair, biased, or inaccurate outcomes. We argue that data collection for AI should be performed in a responsible manner where the quality of the data is thoroughly scrutinized and measured through a systematic set of appropriate metrics. In this paper, we propose a Responsible AI (RAI) methodology designed to guide the data collection with a set of metrics for an iterative in-depth analysis of the factors influencing the quality and reliability} of the generated data. We propose a granular set of measurements to inform on the internal reliability of a dataset and its external stability over time. We validate our approach across nine existing datasets and annotation tasks and four content modalities. This approach impacts the assessment of data robustness used for AI applied in the real world, where diversity of users and content is eminent. Furthermore, it deals with fairness and accountability aspects in data collection by providing systematic and transparent quality analysis for data collections.