mlsanityData to 7 Oct 2026

QueerBench: Quantifying Discrimination in Language Models Toward Queer Identities

arXiv 2406.12399 ↗First posted 18 Jun 2024cs.CL cs.AI cs.CYcited by 5

Not on the map. Why?

Abstract (arXiv)

With the increasing role of Natural Language Processing (NLP) in various applications, challenges concerning bias and stereotype perpetuation are accentuated, which often leads to hate speech and harm. Despite existing studies on sexism and misogyny, issues like homophobia and transphobia remain underexplored and often adopt binary perspectives, putting the safety of LGBTQIA+ individuals at high risk in online spaces. In this paper, we assess the potential harm caused by sentence completions generated by English large language models (LLMs) concerning LGBTQIA+ individuals. This is achieved using QueerBench, our new assessment framework, which employs a template-based approach and a Masked Language Modeling (MLM) task. The analysis indicates that large language models tend to exhibit discriminatory behaviour more frequently towards individuals within the LGBTQIA+ community, reaching a difference gap of 7.2% in the QueerBench score of harmfulness.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 15 of its 40 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

15 of 40 references.

References · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  4. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  5. 2020BERTweet: A pre-trained language model for English Tweetscited by 346
  6. 2019Mitigating Gender Bias in Natural Language Processing: Literature Reviewcited by 287
  7. 2019Toward Gender-Inclusive Coreference Resolutioncited by 84
  8. 2022Theories of "Gender" in NLP Bias Researchcited by 62
  9. 2023SemEval-2023 Task 10: Explainable Detection of Online Sexismcited by 48
  10. 2022Welcome to the Modern World of Pronouns: Identity-Inclusive Natural Language Processing beyond Gendercited by 47
  11. 2023MISGENDERED: Limits of Large Language Models in Understanding Pronounscited by 43
  12. 2021Dataset for Identification of Homophobia and Transophobia in Multilingual YouTube Commentscited by 20
3 more
  1. 2018Evaluating Fairness Metrics in the Presence of Dataset Biascited by 15
  2. 2021Countering Online Hate Speech: An NLP Perspectivecited by 12
  3. 2023On the Origins of Bias in NLP through the Lens of the Jim Codecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 0; 2026: 3.

5 papers.

  1. 2024Toxicity of the Commons: Curating Open-Source Pre-Training Datacited by 7
  2. 2026The Homogenization Problem in LLMs: Towards Meaningful Diversity in AI Safetycited by 1
  3. 2026QueerGen: How LLMs Reflect Societal Norms on Gender and Sexuality in Sentence Completion Taskscited by 1
  4. 2024"Is Hate Lost in Translation?": Evaluation of Multilingual LGBTQIA+ Hate Speech Detectioncited by 0
  5. 2026WinoQueer-NL: Assessing Bias in Dutch Language Models toward LGBTQ+ Identitiescited by 0