# A Systematic Evaluation of Traditional Privacy Policy Analysis Tools Against LLMs

arXiv: https://arxiv.org/abs/2607.17075 · HTML: https://mlsanity.com/paper/2607.17075/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 51 references that are in the map.
- First posted on arXiv 19 Jul 2026.

## Builds on (most cited first, 16 of 16)

- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism](https://mlsanity.com/paper/2407.10457/) (2024)
- [A Thorough Examination of Decoding Methods in the Era of LLMs](https://mlsanity.com/paper/2402.06925/) (2024)
- [Large Language Models Meet NLP: A Survey](https://mlsanity.com/paper/2405.12819/) (2024)
- [Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners](https://mlsanity.com/paper/2305.14825/) (2023)
- [Polisis: Automated Analysis and Presentation of Privacy Policies Using Deep Learning](https://mlsanity.com/paper/1802.02561/) (2018)
- [We Value Your Privacy ... Now Take Some Cookies: Measuring the GDPR's Impact on Web Privacy](https://mlsanity.com/paper/1808.05096/) (2018)
- [Large Language Models: A New Approach for Privacy Policy Analysis at Scale](https://mlsanity.com/paper/2405.20900/) (2024)
- [PoliGraph: Automated Privacy Policy Analysis using Knowledge Graphs (Journal Version)](https://mlsanity.com/paper/2210.06746/) (2022)
- [Empowering Users in Digital Privacy Management through Interactive LLM-Based Agents](https://mlsanity.com/paper/2410.11906/) (2024)
- [Is It a Trap? A Large-scale Empirical Study And Comprehensive Assessment of Online Automated Privacy Policy Generators for Mobile Apps](https://mlsanity.com/paper/2305.03271/) (2023)
- ["You don't need a university degree to comprehend data protection this way": LLM-Powered Interactive Privacy Policy Assessment](https://mlsanity.com/paper/2503.03587/) (2025)
- [{A New Hope}: Contextual Privacy Policies for Mobile Applications and An Approach Toward Automated Generation](https://mlsanity.com/paper/2402.14544/) (2024)
- [Creation and Analysis of an International Corpus of Privacy Laws](https://mlsanity.com/paper/2206.14169/) (2022)
- [Automating Governing Knowledge Commons and Contextual Integrity (GKC-CI) Privacy Policy Annotations with Large Language Models](https://mlsanity.com/paper/2311.02192/) (2023)

## Abstract (arXiv)

The advent of LLMs has significantly changed the research on privacy policy and data compliance analysis by enabling tasks that previously required specialized, domain-specific tools. However, it remains unclear to what extent LLMs can truly replicate the diverse functionalities, and the wide range of methodologies and analysis offered by prior work. In this paper, we conduct the first systematic evaluation of whether off-the-shelf LLMs can replace specialized privacy analysis tools. We study six representative tools spanning three major functionalities: contradiction detection, regulatory compliance analysis, and privacy policy summarization and aggregation, and across three intermediate tasks: structured data extraction using tuples, Semantic Role Labeling (SRL) and manual privacy policy labeling. We compare the performance of two state-of-the-art LLMs (GPT-5.2 and Gemini-2.5 in various configurations) against the tools by directly prompting the models to perform corresponding functionalities and tasks on a custom dataset of 10 privacy policies, allowing us to assess whether off-the-shelf models can produce tool-specific functionalities without further engineering or domain-specific training, major limitations in prior work. Our results show that LLMs consistently match or exceed the capabilities of existing tools across the functionalities. In manual labeling of first-party collection entities, LLMs achieved an average precision of 81.8% and recall of 70.9%, while for labeling of third-party sharing entities, they achieved an average precision of 91.4% and recall of 70.8% compared to the OPP-115 dataset. Overall, our findings indicate that LLMs can effectively perform a broad range of functionalities and tasks in privacy policy and regulation analysis that previously required specialized tools.