mlsanityData to 14 Oct 2026

A Group Fairness Lens for Large Language Models

arXiv 2312.15478 ↗First posted 24 Dec 2023cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

The need to assess LLMs for bias and fairness is critical, with current evaluations often being narrow, missing a broad categorical view. In this paper, we propose evaluating the bias and fairness of LLMs from a group fairness lens using a novel hierarchical schema characterizing diverse social groups. Specifically, we construct a dataset, GFAIR, encapsulating target-attribute combinations across multiple dimensions. Moreover, we introduce statement organization, a new open-ended text generation task, to uncover complex biases in LLMs. Extensive evaluations of popular LLMs reveal inherent safety concerns. To mitigate the biases of LLMs from a group fairness perspective, we pioneer a novel chainof-thought method GF-THINK to mitigate biases of LLMs from a group fairness perspective. Experimental results demonstrate its efficacy in mitigating bias and achieving fairness in LLMs. Our dataset and codes are available at https://github.com/surika/Group-Fairness-LLMs.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 46 of its 90 references that are in the map. First posted on arXiv 24 Dec 2023.

Builds on

46 of 90 references.

References · 46

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  9. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  10. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  11. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  12. 2016Semantics derived automatically from language corpora contain human-like biasescited by 901
34 more
  1. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  2. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  3. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 659
  4. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 541
  5. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
  6. 2018Gender Bias in Coreference Resolutioncited by 442
  7. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 412
  8. 2019On Measuring Social Biases in Sentence Encoderscited by 398
  9. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 380
  10. 2018Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systemscited by 234
  11. 2018Gender Bias in Neural Natural Language Processingcited by 211
  12. 2019Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphologycited by 200
  13. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 185
  14. 2020Social Biases in NLP Models as Barriers for Persons with Disabilitiescited by 182
  15. 2022"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Datasetcited by 162
  16. 2021RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Modelscited by 149
  17. 2022TimeLMs: Diachronic Language Models from Twittercited by 131
  18. 2021Sustainable Modular Debiasing of Language Modelscited by 100
  19. 2020Hate Speech Detection and Racial Bias Mitigation in Social Media based on BERT modelcited by 87
  20. 2020The Radicalization Risks of GPT-3 and Advanced Neural Language Modelscited by 76
  21. 2022TweetNLP: Cutting-Edge Natural Language Processing for Social Mediacited by 71
  22. 2020PowerTransformer: Unsupervised Controllable Revision for Biased Language Correctioncited by 59
  23. 2019Reducing Gender Bias in Word-Level Language Models with a Gender-Equalizing Loss Functioncited by 57
  24. 2023BiasAsker: Measuring the Bias in Conversational AI Systemcited by 53
  25. 2023ROBBIE: Robust Bias Evaluation of Large Generative Language Modelscited by 53
  26. 2023WinoQueer: A Community-in-the-Loop Benchmark for Anti-LGBTQ+ Bias in Large Language Modelscited by 42
  27. 2019A General Framework for Implicit and Explicit Debiasing of Distributional Word Vector Spacescited by 36
  28. 2021Balancing out Bias: Achieving Fairness Through Balanced Trainingcited by 34
  29. 2023Social Bias Probing: Fairness Benchmarking for Language Modelscited by 32
  30. 2021First the worst: Finding better gender translations during beam searchcited by 29
  31. 2022Choose Your Lenses: Flaws in Gender Bias Evaluationcited by 29
  32. 2023On the Independence of Association Bias and Empirical Fairness in Language Modelscited by 29
  33. 2022Text Style Transfer for Bias Mitigation using Masked Language Modelingcited by 28
  34. 2023What does ChatGPT return about human values? Exploring value bias in ChatGPT using a descriptive value theorycited by 19

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 1.

4 papers.

  1. 2024Building Guardrails for Large Language Modelscited by 65
  2. 2024Few-Shot Fairness: Unveiling LLM's Potential for Fairness-Aware Classificationcited by 18
  3. 2024A Survey on Human-AI Collaboration with Large Foundation Modelscited by 7
  4. 2025Towards Large Language Models that Benefit for All: Benchmarking Group Fairness in Reward Modelscited by 2