# Legally Binding but Unfair? Towards Assessing Fairness of Privacy Policies

arXiv: https://arxiv.org/abs/2403.08115 · HTML: https://mlsanity.com/paper/2403.08115/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 95 references that are in the map.
- First posted on arXiv 12 Mar 2024.

## Builds on (most cited first, 26 of 26)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016)
- [Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine](https://mlsanity.com/paper/2311.16452/) (2023)
- [Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes](https://mlsanity.com/paper/1711.08412/) (2017)
- ["I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Dataset](https://mlsanity.com/paper/2205.09209/) (2022)
- [Gender Bias in Machine Translation](https://mlsanity.com/paper/2104.06001/) (2021)
- [Why Fairness Cannot Be Automated: Bridging the Gap Between EU Non-Discrimination Law and AI](https://mlsanity.com/paper/2005.05906/) (2020)
- [Handling Bias in Toxic Speech Detection: A Survey](https://mlsanity.com/paper/2202.00126/) (2022)
- [Polisis: Automated Analysis and Presentation of Privacy Policies Using Deep Learning](https://mlsanity.com/paper/1802.02561/) (2018)
- [DiscoScore: Evaluating Text Generation with BERT and Discourse Coherence](https://mlsanity.com/paper/2201.11176/) (2022)
- [Fairness Perceptions of Algorithmic Decision-Making: A Systematic Review of the Empirical Literature](https://mlsanity.com/paper/2103.12016/) (2021)
- [Fairness in representation: quantifying stereotyping as a representational harm](https://mlsanity.com/paper/1901.09565/) (2019)
- ["There Is Not Enough Information": On the Effects of Explanations on Perceptions of Informational Fairness and Trustworthiness in Automated Decision-Making](https://mlsanity.com/paper/2205.05758/) (2022)
- [The Long Arc of Fairness: Formalisations and Ethical Discourse](https://mlsanity.com/paper/2203.06038/) (2022)
- [word2word: A Collection of Bilingual Lexicons for 3,564 Language Pairs](https://mlsanity.com/paper/1911.12019/) (2019)
- [AI-enabled Automation for Completeness Checking of Privacy Policies](https://mlsanity.com/paper/2106.05688/) (2021)
- [Evaluating Metrics for Bias in Word Embeddings](https://mlsanity.com/paper/2111.07864/) (2021)
- [Maximal Fairness](https://mlsanity.com/paper/2304.06057/) (2023)
- [The Gender-GAP Pipeline: A Gender-Aware Polyglot Pipeline for Gender Characterisation in 55 Languages](https://mlsanity.com/paper/2308.16871/) (2023)
- [How to Make Privacy Policies both GDPR-Compliant and Usable](https://mlsanity.com/paper/1806.06670/) (2018)
- [LLMs grasp morality in concept](https://mlsanity.com/paper/2311.02294/) (2023)

## Built on by (most cited first, 1 of 1)

- [Helping Johnny Make Sense of Privacy Policies with LLMs](https://mlsanity.com/paper/2501.16033/) (2025)

## Abstract (arXiv)

Privacy policies are expected to inform data subjects about their data protection rights and should explain the data controller's data management practices. Privacy policies only fulfill their purpose, if they are correctly interpreted, understood, and trusted by the data subject. This implies that a privacy policy is written in a fair way, e.g., it does not use polarizing terms, does not require a certain education, or does not assume a particular social background. We outline our approach to assessing fairness in privacy policies. We identify from fundamental legal sources and fairness research, how the dimensions informational fairness, representational fairness and ethics / morality are related to privacy policies. We propose options to automatically assess policies in these fairness dimensions, based on text statistics, linguistic methods and artificial intelligence. We conduct initial experiments with German privacy policies to provide evidence that our approach is applicable. Our experiments indicate that there are issues in all three dimensions of fairness. This is important, as future privacy policies may be used in a corpus for legal artificial intelligence models.