# Defining and Evaluating Fair Natural Language Generation

arXiv: https://arxiv.org/abs/2008.01548 · HTML: https://mlsanity.com/paper/2008.01548/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jul 2020.

## Built on by (most cited first, 17 of 17)

- [Societal Biases in Language Generation: Progress and Challenges](https://mlsanity.com/paper/2105.04054/) (2021)
- [RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Models](https://mlsanity.com/paper/2106.03521/) (2021)
- [Trustworthy AI: A Computational Perspective](https://mlsanity.com/paper/2107.06641/) (2021)
- [Theories of "Gender" in NLP Bias Research](https://mlsanity.com/paper/2205.02526/) (2022)
- [Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory](https://mlsanity.com/paper/2305.14889/) (2023)
- [CHBias: Bias Evaluation and Mitigation of Chinese Conversational Language Models](https://mlsanity.com/paper/2305.11262/) (2023)
- [Stereotypes and Smut: The (Mis)representation of Non-cisgender Identities by Text-to-Image Models](https://mlsanity.com/paper/2305.17072/) (2023)
- [Equi-Tuning: Group Equivariant Fine-Tuning of Pretrained Models](https://mlsanity.com/paper/2210.06475/) (2022)
- [Learning to Generate Equitable Text in Dialogue from Biased Training Data](https://mlsanity.com/paper/2307.04303/) (2023)
- [Generating Full Length Wikipedia Biographies: The Impact of Gender Bias on the Retrieval-Based Generation of Women Biographies](https://mlsanity.com/paper/2204.05879/) (2022)
- [An Analysis of the Effects of Decoding Algorithms on Fairness in Open-Ended Language Generation](https://mlsanity.com/paper/2210.03826/) (2022)
- [Towards Robust NLG Bias Evaluation with Syntactically-diverse Prompts](https://mlsanity.com/paper/2212.01700/) (2022)
- [FairPair: A Robust Evaluation of Biases in Language Models through Paired Perturbations](https://mlsanity.com/paper/2404.06619/) (2024)
- [Entity-Based Evaluation of Political Bias in Automatic Summarization](https://mlsanity.com/paper/2305.02321/) (2023)
- [From Charts to Fair Narratives: Uncovering and Mitigating Geo-Economic Biases in Chart-to-Text](https://mlsanity.com/paper/2508.09450/) (2025)
- [A Note on Bias to Complete](https://mlsanity.com/paper/2402.11710/) (2024)
- [Democratizing Ethical Assessment of Natural Language Generation Models](https://mlsanity.com/paper/2207.10576/) (2022)

## Abstract (arXiv)

Our work focuses on the biases that emerge in the natural language generation (NLG) task of sentence completion. In this paper, we introduce a framework of fairness for NLG followed by an evaluation of gender biases in two state-of-the-art language models. Our analysis provides a theoretical formulation for biases in NLG and empirical evidence that existing language generation models embed gender bias.