# Democratizing Ethical Assessment of Natural Language Generation Models

arXiv: https://arxiv.org/abs/2207.10576 · HTML: https://mlsanity.com/paper/2207.10576/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 42 references that are in the map.
- First posted on arXiv 30 Jun 2022.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation](https://mlsanity.com/paper/2101.11718/) (2021)
- [Towards Understanding and Mitigating Social Biases in Language Models](https://mlsanity.com/paper/2106.13219/) (2021)
- [Predicting the Type and Target of Offensive Posts in Social Media](https://mlsanity.com/paper/1902.09666/) (2019)
- [Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://mlsanity.com/paper/2111.07997/) (2021)
- [Plan-And-Write: Towards Better Automatic Storytelling](https://mlsanity.com/paper/1811.05701/) (2018)
- [Adversarial Robustness Toolbox v1.0.0](https://mlsanity.com/paper/1807.01069/) (2018)
- [Societal Biases in Language Generation: Progress and Challenges](https://mlsanity.com/paper/2105.04054/) (2021)
- [What Does it Mean for a Language Model to Preserve Privacy?](https://mlsanity.com/paper/2202.05520/) (2022)
- [Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets](https://mlsanity.com/paper/2106.10328/) (2021)
- [RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Models](https://mlsanity.com/paper/2106.03521/) (2021)
- [Deceiving Google's Perspective API Built for Detecting Toxic Comments](https://mlsanity.com/paper/1702.08138/) (2017)
- [Neural Machine Translation: A Review and Survey](https://mlsanity.com/paper/1912.02047/) (2019)
- [Pretraining-Based Natural Language Generation for Text Summarization](https://mlsanity.com/paper/1902.09243/) (2019)
- [Defining and Evaluating Fair Natural Language Generation](https://mlsanity.com/paper/2008.01548/) (2020)
- [Software-Based Dialogue Systems: Survey, Taxonomy and Challenges](https://mlsanity.com/paper/2106.10901/) (2021)
- [CLARA: Clinical Report Auto-completion](https://mlsanity.com/paper/2002.11701/) (2020)

## Abstract (arXiv)

Natural language generation models are computer systems that generate coherent language when prompted with a sequence of words as context. Despite their ubiquity and many beneficial applications, language generation models also have the potential to inflict social harms by generating discriminatory language, hateful speech, profane content, and other harmful material. Ethical assessment of these models is therefore critical. But it is also a challenging task, requiring an expertise in several specialized domains, such as computational linguistics and social justice. While significant strides have been made by the research community in this domain, accessibility of such ethical assessments to the wider population is limited due to the high entry barriers. This article introduces a new tool to democratize and standardize ethical assessment of natural language generation models: Tool for Ethical Assessment of Language generation models (TEAL), a component of Credo AI Lens, an open-source assessment framework.