mlsanityData to 7 Oct 2026

Casual Conversations v2: Designing a large consent-driven dataset to measure algorithmic bias and robustness

arXiv 2211.05809 ↗First posted 10 Nov 2022cs.CV cs.AI cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Developing robust and fair AI systems require datasets with comprehensive set of labels that can help ensure the validity and legitimacy of relevant measurements. Recent efforts, therefore, focus on collecting person-related datasets that have carefully selected labels, including sensitive characteristics, and consent forms in place to use those attributes for model testing and development. Responsible data collection involves several stages, including but not limited to determining use-case scenarios, selecting categories (annotations) such that the data are fit for the purpose of measuring algorithmic bias for subgroups and most importantly ensure that the selected categories/subcategories are robust to regional diversities and inclusive of as many subgroups as possible. Meta, in a continuation of our efforts to measure AI algorithmic bias and robustness (https://ai.facebook.com/blog/shedding-light-on-fairness-in-ai-with-a-new-data-set), is working on collecting a large consent-driven dataset with a comprehensive list of categories. This paper describes our proposed design of such categories and subcategories for Casual Conversations v2.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 39 of its 143 references that are in the map. First posted on arXiv 10 Nov 2022.

Builds on

39 of 143 references.

References · 39

  1. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  2. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  3. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  4. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  5. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  6. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  7. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  8. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  9. 2017AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wildcited by 490
  10. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  11. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377
  12. 2019Mitigating Gender Bias in Natural Language Processing: Literature Reviewcited by 287
27 more
  1. 2019Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigationcited by 252
  2. 2020Fairness without Demographics through Adversarially Reweighted Learningcited by 251
  3. 2019Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFacecited by 222
  4. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
  5. 2018Reducing Gender Bias in Abusive Language Detectioncited by 152
  6. 2017Beyond Parity: Fairness Objectives for Collaborative Filteringcited by 93
  7. 2021A Survey on Bias in Visual Datasetscited by 84
  8. 2021Towards Measuring Fairness in AI: the Casual Conversations Datasetcited by 80
  9. 2019Diversity in Facescited by 77
  10. 2019Predictive Inequity in Object Detectioncited by 77
  11. 2021Fairness via Representation Neutralizationcited by 63
  12. 2020PowerTransformer: Unsupervised Controllable Revision for Biased Language Correctioncited by 59
  13. 2022Meta Balanced Network for Fair Face Recognitioncited by 53
  14. 2021Quantifying Bias in Automatic Speech Recognitioncited by 50
  15. 2020Analysis of Gender Inequality In Face Recognition Accuracycited by 47
  16. 2021Learning Fair Classifiers with Partially Annotated Group Labelscited by 40
  17. 2021Gendered Differences in Face Recognition Accuracy Explained by Hairstyles, Makeup, and Facial Morphologycited by 32
  18. 2018An Experimental Evaluation of Covariates Effects on Unconstrained Face Verificationcited by 28
  19. 2021Towards Measuring Fairness in Speech Recognition: Casual Conversations Dataset Transcriptionscited by 26
  20. 2018Understanding Unequal Gender Classification Accuracy from Face Imagescited by 25
  21. 2017Deep Residual Bidir-LSTM for Human Activity Recognition Using Wearable Sensorscited by 24
  22. 2022Fairness Indicators for Systematic Assessments of Visual Feature Extractorscited by 24
  23. 2018AI Fairness for People with Disabilities: Point of Viewcited by 16
  24. 2019Does Face Recognition Accuracy Get Better With Age? Deep Face Matchers Say Nocited by 16
  25. 2020An Ethical Highlighter for People-Centric Dataset Creationcited by 12
  26. 2022Towards Equal Opportunity Fairness through Adversarial Learningcited by 8
  27. 2022Color Invariant Skin Segmentationcited by 3

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 2.

5 papers.

  1. 2023FACET: Fairness in Computer Vision Evaluation Benchmarkcited by 58
  2. 2023Survey of Social Bias in Vision-Language Modelscited by 26
  3. 2024The Bias of Harmful Label Associations in Vision-Language Modelscited by 5
  4. 2023The Casual Conversations v2 Datasetcited by 4
  5. 2024FairLENS: Assessing Fairness in Law Enforcement Speech Recognitioncited by 0