mlsanityData to 7 Oct 2026

Towards WinoQueer: Developing a Benchmark for Anti-Queer Bias in Large Language Models

arXiv 2206.11484 ↗First posted 23 Jun 2022cs.CL cs.CYcited by 8

Not on the map. Why?

Abstract (arXiv)

This paper presents exploratory work on whether and to what extent biases against queer and trans people are encoded in large language models (LLMs) such as BERT. We also propose a method for reducing these biases in downstream tasks: finetuning the models on data written by and/or about queer people. To measure anti-queer bias, we introduce a new benchmark dataset, WinoQueer, modeled after other bias-detection benchmarks but addressing homophobic and transphobic biases. We found that BERT shows significant homophobic bias, but this bias can be mostly mitigated by finetuning BERT on a natural language corpus written by members of the LGBTQ+ community.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 15 of its 32 references that are in the map. First posted on arXiv 23 Jun 2022.

Builds on

15 of 32 references.

References · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  3. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  4. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  5. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  6. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  7. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  8. 2018Gender Bias in Coreference Resolutioncited by 438
  9. 2019On Measuring Social Biases in Sentence Encoderscited by 397
  10. 2019Racial Bias in Hate Speech and Abusive Language Detection Datasetscited by 208
  11. 2019BERT for Coreference Resolution: Baselines and Analysiscited by 125
  12. 2019Toward Gender-Inclusive Coreference Resolutioncited by 84
3 more
  1. 2021Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metricscited by 83
  2. 2021Fairness for Unobserved Characteristics: Insights from Technological Impacts on Queer Communitiescited by 40
  3. 2017Like trainer, like bot? Inheritance of bias in algorithmic content moderationcited by 34

Built on by

Cited by year

Citing papers per year: 2023: 7; 2024: 1.

8 papers.

  1. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  2. 2023Stay on topic with Classifier-Free Guidancecited by 90
  3. 2023"I'm fully who I am": Towards Centering Transgender and Non-Binary Voices to Measure Biases in Open Language Generationcited by 56
  4. 2023A Short Survey of Viewing Large Language Models in Legal Aspectcited by 51
  5. 2023WinoQueer: A Community-in-the-Loop Benchmark for Anti-LGBTQ+ Bias in Large Language Modelscited by 42
  6. 2023Scaling Laws Do Not Scalecited by 13
  7. 2024Exploring the Nexus of Large Language Models and Legal Systems: A Short Surveycited by 6
  8. 2023In What Languages are Generative Language Models the Most Formal? Analyzing Formality Distribution across Languagescited by 2