# Challenges in Annotating Datasets to Quantify Bias in Under-represented Society

arXiv: https://arxiv.org/abs/2309.08624 · HTML: https://mlsanity.com/paper/2309.08624/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 45 references that are in the map.
- First posted on arXiv 11 Sep 2023.

## Builds on (most cited first, 24 of 24)

- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Certifying and removing disparate impact](https://mlsanity.com/paper/1412.3756/) (2014)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation](https://mlsanity.com/paper/2101.11718/) (2021)
- [On Measuring Social Biases in Sentence Encoders](https://mlsanity.com/paper/1903.10561/) (2019)
- [Evaluating Gender Bias in Machine Translation](https://mlsanity.com/paper/1906.00591/) (2019)
- [Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems](https://mlsanity.com/paper/1805.04508/) (2018)
- ["I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Dataset](https://mlsanity.com/paper/2205.09209/) (2022)
- [UnQovering Stereotyping Biases via Underspecified Questions](https://mlsanity.com/paper/2010.02428/) (2020)
- [On Measuring and Mitigating Biased Inferences of Word Embeddings](https://mlsanity.com/paper/1908.09369/) (2019)
- [An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models](https://mlsanity.com/paper/2110.08527/) (2021)
- [Challenges in Automated Debiasing for Toxic Language Detection](https://mlsanity.com/paper/2102.00086/) (2021)
- [Multi-Dimensional Gender Bias Classification](https://mlsanity.com/paper/2005.00614/) (2020)
- [OSCaR: Orthogonal Subspace Correction and Rectification of Biases in Word Embeddings](https://mlsanity.com/paper/2007.00049/) (2020)
- [Towards Identifying Social Bias in Dialog Systems: Frame, Datasets, and Benchmarks](https://mlsanity.com/paper/2202.08011/) (2022)
- [Towards Understanding Gender Bias in Relation Extraction](https://mlsanity.com/paper/1911.03642/) (2019)
- [Fair and Argumentative Language Modeling for Computational Argumentation](https://mlsanity.com/paper/2204.04026/) (2022)
- [Effectiveness of Debiasing Techniques: An Indigenous Qualitative Analysis](https://mlsanity.com/paper/2304.11094/) (2023)

## Built on by (most cited first, 1 of 1)

- [Tackling Bias in Pre-trained Language Models: Current Trends and Under-represented Societies](https://mlsanity.com/paper/2312.01509/) (2023)

## Abstract (arXiv)

Recent advances in artificial intelligence, including the development of highly sophisticated large language models (LLM), have proven beneficial in many real-world applications. However, evidence of inherent bias encoded in these LLMs has raised concerns about equity. In response, there has been an increase in research dealing with bias, including studies focusing on quantifying bias and developing debiasing techniques. Benchmark bias datasets have also been developed for binary gender classification and ethical/racial considerations, focusing predominantly on American demographics. However, there is minimal research in understanding and quantifying bias related to under-represented societies. Motivated by the lack of annotated datasets for quantifying bias in under-represented societies, we endeavoured to create benchmark datasets for the New Zealand (NZ) population. We faced many challenges in this process, despite the availability of three annotators. This research outlines the manual annotation process, provides an overview of the challenges we encountered and lessons learnt, and presents recommendations for future research.