mlsanityData to 7 Oct 2026

GenderAlign: An Alignment Dataset for Mitigating Gender Bias in Large Language Models

arXiv 2406.13925 ↗First posted 20 Jun 2024cs.CL cs.AIcited by 18

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Large Language Models (LLMs) are prone to generating content that exhibits gender biases, raising significant ethical concerns. Alignment, the process of fine-tuning LLMs to better align with desired behaviors, is recognized as an effective approach to mitigate gender biases. Although proprietary LLMs have made significant strides in mitigating gender bias, their alignment datasets are not publicly available. The commonly used and publicly available alignment dataset, HH-RLHF, still exhibits gender bias to some extent. There is a lack of publicly available alignment datasets specifically designed to address gender bias. Hence, we developed a new dataset named GenderAlign, aiming at mitigating a comprehensive set of gender biases in LLMs. This dataset comprises 8k single-turn dialogues, each paired with a "chosen" and a "rejected" response. Compared to the "rejected" responses, the "chosen" responses demonstrate lower levels of gender bias and higher quality. Furthermore, we categorized the gender biases in the "rejected" responses of GenderAlign into 4 principal categories. The experimental results show the effectiveness of GenderAlign in reducing gender bias in LLMs.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 23 of its 40 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

23 of 40 references.

References · 23

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  9. 2021Ethical and social risks of harm from Language Modelscited by 957
  10. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  11. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  12. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
11 more
  1. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  2. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  3. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  4. 2018Gender Bias in Coreference Resolutioncited by 438
  5. 2023"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letterscited by 196
  6. 2021What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpuscited by 79
  7. 2020"Call me sexist, but...": Revisiting Sexism Detection Using Psychological Scales and Adversarial Samplescited by 48
  8. 2024Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Datacited by 33
  9. 2023CORGI-PM: A Chinese Corpus For Gender Bias Probing and Mitigationcited by 19
  10. 2023Towards Auditing Large Language Models: Improving Text-based Stereotype Detectioncited by 12
  11. 2020Automatic Detection of Sexist Statements Commonly Used at the Workplacecited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 9; 2026: 6.

18 papers.

Sort:

By area:

  1. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 175
  2. 2024Multimodal Alignment and Fusion: A Surveycited by 73
  3. 2025Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLMcited by 8
  4. 2025Evaluating the Effect of Retrieval Augmentation on Social Biasescited by 7
  5. 2024The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Modelscited by 5
  6. 2026When Agents See Humans as the Outgroup: Belief-Dependent Bias in LLM-Powered Agentscited by 5
  7. 2025BiasFreeBench: a Benchmark for Mitigating Bias in Large Language Model Responsescited by 2
  8. 2025Large Language Models Develop Novel Social Biases Through Adaptive Explorationcited by 2
  9. 2026Alignment Reduces Expressed but Not Encoded Gender Bias: A Unified Framework and Studycited by 2
  10. 2024Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyondcited by 1
  11. 2025Thinking About Thinking: Evaluating Reasoning in Post-Trained Language Modelscited by 1
  12. 2025Evaluating Implicit Biases in LLM Reasoning through Logic Grid Puzzlescited by 1
6 more
  1. 2026The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotationcited by 1
  2. 2025Auto-Search and Refinement: An Automated Framework for Gender Bias Mitigation in Large Language Modelscited by 0
  3. 2025The Fair Game: Auditing & Debiasing AI Algorithms Over Timecited by 0
  4. 2025Textual Data Bias Detection and Mitigation -- An Extensible Pipeline with Experimental Evaluationcited by 0
  5. 2026Analysing LLM Persona Generation and Fairness Interpretation in Polarised Geopolitical Contextscited by 0
  6. 2026Neutrality Bites: Gender Representation in AI-Generated Animal Storiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025More Women, Same Stereotypes: Unpacking the Gender Bias Paradox in Large Language Models
  2. 2023Gender bias and stereotypes in Large Language Models
  3. 2024Disclosure and Mitigation of Gender Bias in LLMs
  4. 2025Mitigating Gender Bias via Fostering Exploratory Thinking in LLMs
  5. 2025EuroGEST: Investigating gender stereotypes in multilingual language models
  6. 2024Robustness and Confounders in the Demographic Alignment of LLMs with Human Perceptions of Offensiveness
  7. 2025Detection, Classification, and Mitigation of Gender Bias in Large Language Models
  8. 2024Uncovering Factor Level Preferences to Improve Human-Model Alignment