# Large Language Models: A New Approach for Privacy Policy Analysis at Scale

arXiv: https://arxiv.org/abs/2405.20900 · HTML: https://mlsanity.com/paper/2405.20900/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 49 references that are in the map.
- First posted on arXiv 31 May 2024.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data](https://mlsanity.com/paper/2302.12822/) (2023)
- [Question Answering for Privacy Policies: Combining Computational and Legal Perspectives](https://mlsanity.com/paper/1911.00841/) (2019)
- [Polisis: Automated Analysis and Presentation of Privacy Policies Using Deep Learning](https://mlsanity.com/paper/1802.02561/) (2018)
- [Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future](https://mlsanity.com/paper/2206.02280/) (2022)
- [PolicyGPT: Automated Analysis of Privacy Policies with Large Language Models](https://mlsanity.com/paper/2309.10238/) (2023)
- [ChatGPT for Suicide Risk Assessment on Social Media: Quantitative Evaluation of Model Performance, Potentials and Limitations](https://mlsanity.com/paper/2306.09390/) (2023)
- [Modeling Language Vagueness in Privacy Policies using Deep Neural Networks](https://mlsanity.com/paper/1805.10393/) (2018)

## Built on by (most cited first, 14 of 14)

- [Privacy Policy Analysis through Prompt Engineering for LLMs](https://mlsanity.com/paper/2409.14879/) (2024)
- [Helping Johnny Make Sense of Privacy Policies with LLMs](https://mlsanity.com/paper/2501.16033/) (2025)
- [PolicyLR: A Logic Representation For Privacy Policies](https://mlsanity.com/paper/2408.14830/) (2024)
- [Word-level Annotation of GDPR Transparency Compliance in Privacy Policies using Large Language Models](https://mlsanity.com/paper/2503.10727/) (2025)
- [Let's Measure the Elephant in the Room: Facilitating Personalized Automated Analysis of Privacy Policies at Scale](https://mlsanity.com/paper/2507.14214/) (2025)
- [AudAgent: Automated Auditing of Privacy Policy Compliance in AI Agents](https://mlsanity.com/paper/2511.07441/) (2025)
- [AI-driven Personalized Privacy Assistants: a Systematic Literature Review](https://mlsanity.com/paper/2502.07693/) (2025)
- [Interpretation Gaps in LLM-Assisted Comprehension of Privacy Documents](https://mlsanity.com/paper/2503.12225/) (2025)
- [Generating Privacy Stories From Software Documentation](https://mlsanity.com/paper/2506.23014/) (2025)
- [An LLM-enabled semantic-centric framework to consume privacy policies](https://mlsanity.com/paper/2509.01716/) (2025)
- [LADFA: A Framework of Using Large Language Models and Retrieval-Augmented Generation for Personal Data Flow Analysis in Privacy Policies](https://mlsanity.com/paper/2601.10413/) (2026)
- [A Systematic Evaluation of Traditional Privacy Policy Analysis Tools Against LLMs](https://mlsanity.com/paper/2607.17075/) (2026)
- [Enabling Multilingual Privacy Policy Audits: Large-Scale Analysis of Spanish Mobile Apps](https://mlsanity.com/paper/2607.18424/) (2026)
- [Privacy Washing: Detecting Internal Contradictions in Privacy Policies](https://mlsanity.com/paper/2609.02055/) (2026)

## Abstract (arXiv)

The number and dynamic nature of web and mobile applications presents significant challenges for assessing their compliance with data protection laws. In this context, symbolic and statistical Natural Language Processing (NLP) techniques have been employed for the automated analysis of these systems' privacy policies. However, these techniques typically require labor-intensive and potentially error-prone manually annotated datasets for training and validation. This research proposes the application of Large Language Models (LLMs) as an alternative for effectively and efficiently extracting privacy practices from privacy policies at scale. Particularly, we leverage well-known LLMs such as ChatGPT and Llama 2, and offer guidance on the optimal design of prompts, parameters, and models, incorporating advanced strategies such as few-shot learning. We further illustrate its capability to detect detailed and varied privacy practices accurately. Using several renowned datasets in the domain as a benchmark, our evaluation validates its exceptional performance, achieving an F1 score exceeding 93%. Besides, it does so with reduced costs, faster processing times, and fewer technical knowledge requirements. Consequently, we advocate for LLM-based solutions as a sound alternative to traditional NLP techniques for the automated analysis of privacy policies at scale.