# Fairness Certification for Natural Language Processing and Large Language Models

arXiv: https://arxiv.org/abs/2401.01262 · HTML: https://mlsanity.com/paper/2401.01262/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 75 references that are in the map.
- First posted on arXiv 2 Jan 2024.

## Builds on (most cited first, 30 of 32)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016)
- [Algorithmic decision making and the cost of fairness](https://mlsanity.com/paper/1701.08230/) (2017)
- [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://mlsanity.com/paper/1906.08976/) (2019)
- [Evaluating Gender Bias in Machine Translation](https://mlsanity.com/paper/1906.00591/) (2019)
- [A comparative study of fairness-enhancing interventions in machine learning](https://mlsanity.com/paper/1802.04422/) (2018)
- [Counterfactual Fairness in Text Classification through Robustness](https://mlsanity.com/paper/1809.10610/) (2018)
- [Why Is My Classifier Discriminatory?](https://mlsanity.com/paper/1805.12002/) (2018)
- [On the Apparent Conflict Between Individual and Group Fairness](https://mlsanity.com/paper/1912.06883/) (2019)
- [Natural Language Processing: State of The Art, Current Trends and Challenges](https://mlsanity.com/paper/1708.05148/) (2017)
- [Racial Disparity in Natural Language Processing: A Case Study of Social Media African-American English](https://mlsanity.com/paper/1707.00061/) (2017)
- [Post-processing for Individual Fairness](https://mlsanity.com/paper/2110.13796/) (2021)
- ["I'm fully who I am": Towards Centering Transgender and Non-Binary Voices to Measure Biases in Open Language Generation](https://mlsanity.com/paper/2305.09941/) (2023)
- [Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem](https://mlsanity.com/paper/2310.02521/) (2023)
- [Residual Unfairness in Fair Machine Learning from Prejudiced Data](https://mlsanity.com/paper/1806.02887/) (2018)
- [How Different Groups Prioritize Ethical Values for Responsible AI](https://mlsanity.com/paper/2205.07722/) (2022)
- [Auditing Black-box Models for Indirect Influence](https://mlsanity.com/paper/1602.07043/) (2016)
- [SenSeI: Sensitive Set Invariance for Enforcing Individual Fairness](https://mlsanity.com/paper/2006.14168/) (2020)
- [Unpacking the Interdependent Systems of Discrimination: Ableist Bias in NLP Systems through an Intersectional Lens](https://mlsanity.com/paper/2110.00521/) (2021)
- [On the Independence of Association Bias and Empirical Fairness in Language Models](https://mlsanity.com/paper/2304.10153/) (2023)
- [Fairness Perceptions of Algorithmic Decision-Making: A Systematic Review of the Empirical Literature](https://mlsanity.com/paper/2103.12016/) (2021)
- [Fairness in the Eyes of the Data: Certifying Machine-Learning Models](https://mlsanity.com/paper/2009.01534/) (2020)
- [AI Certification: Advancing Ethical Practice by Reducing Information Asymmetries](https://mlsanity.com/paper/2105.10356/) (2021)
- [Multi-dimensional discrimination in Law and Machine Learning -- A comparative overview](https://mlsanity.com/paper/2302.05995/) (2023)

## Built on by (most cited first, 1 of 1)

- [Shh, don't say that! Domain Certification in LLMs](https://mlsanity.com/paper/2502.19320/) (2025)

## Abstract (arXiv)

Natural Language Processing (NLP) plays an important role in our daily lives, particularly due to the enormous progress of Large Language Models (LLM). However, NLP has many fairness-critical use cases, e.g., as an expert system in recruitment or as an LLM-based tutor in education. Since NLP is based on human language, potentially harmful biases can diffuse into NLP systems and produce unfair results, discriminate against minorities or generate legal issues. Hence, it is important to develop a fairness certification for NLP approaches. We follow a qualitative research approach towards a fairness certification for NLP. In particular, we have reviewed a large body of literature on algorithmic fairness, and we have conducted semi-structured expert interviews with a wide range of experts from that area. We have systematically devised six fairness criteria for NLP, which can be further refined into 18 sub-categories. Our criteria offer a foundation for operationalizing and testing processes to certify fairness, both from the perspective of the auditor and the audited organization.