mlsanityData to 14 Oct 2026

Multilingual Holistic Bias: Extending Descriptors and Patterns to Unveil Demographic Biases in Languages at Scale

arXiv 2305.13198 ↗First posted 22 May 2023cs.CLcited by 24

Not on the map. Why?

Abstract (arXiv)

We introduce a multilingual extension of the HOLISTICBIAS dataset, the largest English template-based taxonomy of textual people references: MULTILINGUALHOLISTICBIAS. This extension consists of 20,459 sentences in 50 languages distributed across all 13 demographic axes. Source sentences are built from combinations of 118 demographic descriptors and three patterns, excluding nonsensical combinations. Multilingual translations include alternatives for gendered languages that cover gendered translations when there is ambiguity in English. Our benchmark is intended to uncover demographic imbalances and be the tool to quantify mitigations towards them. Our initial findings show that translation quality for EN-to-XX translations is an average of 8 spBLEU better when evaluating with the masculine human reference compared to feminine. In the opposite direction, XX-to-EN, we compare the robustness of the model when the source input only differs in gender (masculine or feminine) and masculine translations are an average of almost 4 spBLEU better than feminine. When embedding sentences to a joint multilingual sentence representations space, we find that for most languages masculine translations are significantly closer to the English neutral sentences when embedded.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 23 of its 27 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

23 of 27 references.

References · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  3. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  4. 2020Language-agnostic BERT Sentence Embeddingcited by 627
  5. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  6. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 541
  7. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 412
  8. 2019On Measuring Social Biases in Sentence Encoderscited by 398
  9. 2019Measuring Bias in Contextualized Word Representationscited by 276
  10. 2019Evaluating Gender Bias in Machine Translationcited by 249
  11. 2020Measuring and Reducing Gendered Correlations in Pre-trained Modelscited by 224
  12. 2022"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Datasetcited by 162
11 more
  1. 2021Gender Bias in Machine Translationcited by 117
  2. 2021Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translationcited by 62
  3. 2017Learning Joint Multilingual Sentence Representations with Neural Machine Translationcited by 52
  4. 2022Perturbation Augmentation for Fairer NLPcited by 52
  5. 2022Bitext Mining Using Distilled Sentence Representations for Low-Resource Languagescited by 48
  6. 2022Towards Opening the Black Box of Neural Machine Translation: Source and Target Interpretations of the Transformercited by 45
  7. 2022AugLy: Data Augmentations for Robustnesscited by 34
  8. 2021Gender Bias Amplification During Speed-Quality Optimization in Neural Machine Translationcited by 28
  9. 2022Toxicity in Multilingual Machine Translation at Scalecited by 26
  10. 2021Investigating Failures of Automatic Translation in the Case of Unambiguous Gendercited by 20
  11. 2020Gender Bias in Multilingual Neural Machine Translation: The Architecture Matterscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 7; 2025: 8; 2026: 4.

24 papers.

Sort:

By area:

  1. 2023Large Language Model Alignment: A Surveycited by 243
  2. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 202
  3. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  4. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 45
  5. 2023FFT: Towards Harmlessness Evaluation and Analysis for LLMs with Factuality, Fairness, Toxicitycited by 25
  6. 2024From One to Many: Expanding the Scope of Toxicity Mitigation in Language Modelscited by 18
  7. 2024What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine Translation with a Human-centered Studycited by 15
  8. 2025Digital Player: Evaluating Large Language Models based Human-like Agent in Gamescited by 6
  9. 2024MiTTenS: A Dataset for Evaluating Gender Mistranslationcited by 5
  10. 2024GATE X-E : A Challenge Set for Gender-Fair Translations from Weakly-Gendered Languagescited by 4
  11. 2023The Gender-GAP Pipeline: A Gender-Aware Polyglot Pipeline for Gender Characterisation in 55 Languagescited by 3
  12. 2023Gender-specific Machine Translation with Large Language Modelscited by 3
12 more
  1. 2025Breaking Language Barriers or Reinforcing Bias? A Study of Gender and Racial Disparities in Multilingual Contrastive Vision Language Modelscited by 2
  2. 2025Social Bias in Multilingual Language Models: A Surveycited by 2
  3. 2025Camellia: Benchmarking Cultural Biases in LLMs for Asian Languagescited by 2
  4. 2025What Triggers my Model? Contrastive Explanations Inform Gender Choices by Translation Modelscited by 2
  5. 2026Measuring Social Bias in Vision-Language Models with Face-Only Counterfactuals from Real Photoscited by 2
  6. 2024Beyond Binary Gender: Evaluating Gender-Inclusive Machine Translation with Ambiguous Attitude Wordscited by 1
  7. 2025Theories of "Sexuality" in Natural Language Processing Bias Researchcited by 1
  8. 2026QuantiBias: Benchmarking Quantization-Induced Bias in LLMscited by 1
  9. 2024Towards Massive Multilingual Holistic Biascited by 0
  10. 2025Probing Social Identity Bias in Chinese LLMs with Gendered Pronouns and Social Groupscited by 0
  11. 2026Gender Bias in MT for a Genderless Language: New Benchmarks for Basquecited by 0
  12. 2026Personas Differ from Native-Language Generation: Language Pathways Shape LLM Interpersonal Advicecited by 0