# Casual Conversations v2: Designing a large consent-driven dataset to measure algorithmic bias and robustness

arXiv: https://arxiv.org/abs/2211.05809 · HTML: https://mlsanity.com/paper/2211.05809/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 143 references that are in the map.
- First posted on arXiv 10 Nov 2022.

## Builds on (most cited first, 30 of 39)

- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://mlsanity.com/paper/1911.03891/) (2019)
- [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://mlsanity.com/paper/1906.08976/) (2019)
- [Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation](https://mlsanity.com/paper/1911.11834/) (2019)
- [Fairness without Demographics through Adversarially Reweighted Learning](https://mlsanity.com/paper/2006.13114/) (2020)
- [Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace](https://mlsanity.com/paper/1910.04855/) (2019)
- [XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation](https://mlsanity.com/paper/2004.01401/) (2020)
- [Reducing Gender Bias in Abusive Language Detection](https://mlsanity.com/paper/1808.07231/) (2018)
- [Beyond Parity: Fairness Objectives for Collaborative Filtering](https://mlsanity.com/paper/1705.08804/) (2017)
- [A Survey on Bias in Visual Datasets](https://mlsanity.com/paper/2107.07919/) (2021)
- [Towards Measuring Fairness in AI: the Casual Conversations Dataset](https://mlsanity.com/paper/2104.02821/) (2021)
- [Diversity in Faces](https://mlsanity.com/paper/1901.10436/) (2019)
- [Predictive Inequity in Object Detection](https://mlsanity.com/paper/1902.11097/) (2019)
- [Fairness via Representation Neutralization](https://mlsanity.com/paper/2106.12674/) (2021)
- [PowerTransformer: Unsupervised Controllable Revision for Biased Language Correction](https://mlsanity.com/paper/2010.13816/) (2020)
- [Meta Balanced Network for Fair Face Recognition](https://mlsanity.com/paper/2205.06548/) (2022)
- [Quantifying Bias in Automatic Speech Recognition](https://mlsanity.com/paper/2103.15122/) (2021)
- [Analysis of Gender Inequality In Face Recognition Accuracy](https://mlsanity.com/paper/2002.00065/) (2020)
- [Learning Fair Classifiers with Partially Annotated Group Labels](https://mlsanity.com/paper/2111.14581/) (2021)
- [Gendered Differences in Face Recognition Accuracy Explained by Hairstyles, Makeup, and Facial Morphology](https://mlsanity.com/paper/2112.14656/) (2021)
- [An Experimental Evaluation of Covariates Effects on Unconstrained Face Verification](https://mlsanity.com/paper/1808.05508/) (2018)

## Built on by (most cited first, 5 of 5)

- [FACET: Fairness in Computer Vision Evaluation Benchmark](https://mlsanity.com/paper/2309.00035/) (2023)
- [Survey of Social Bias in Vision-Language Models](https://mlsanity.com/paper/2309.14381/) (2023)
- [The Bias of Harmful Label Associations in Vision-Language Models](https://mlsanity.com/paper/2402.07329/) (2024)
- [The Casual Conversations v2 Dataset](https://mlsanity.com/paper/2303.04838/) (2023)
- [FairLENS: Assessing Fairness in Law Enforcement Speech Recognition](https://mlsanity.com/paper/2405.13166/) (2024)

## Abstract (arXiv)

Developing robust and fair AI systems require datasets with comprehensive set of labels that can help ensure the validity and legitimacy of relevant measurements. Recent efforts, therefore, focus on collecting person-related datasets that have carefully selected labels, including sensitive characteristics, and consent forms in place to use those attributes for model testing and development. Responsible data collection involves several stages, including but not limited to determining use-case scenarios, selecting categories (annotations) such that the data are fit for the purpose of measuring algorithmic bias for subgroups and most importantly ensure that the selected categories/subcategories are robust to regional diversities and inclusive of as many subgroups as possible. Meta, in a continuation of our efforts to measure AI algorithmic bias and robustness (https://ai.facebook.com/blog/shedding-light-on-fairness-in-ai-with-a-new-data-set), is working on collecting a large consent-driven dataset with a comprehensive list of categories. This paper describes our proposed design of such categories and subcategories for Casual Conversations v2.