# Helping Johnny Make Sense of Privacy Policies with LLMs

arXiv: https://arxiv.org/abs/2501.16033 · HTML: https://mlsanity.com/paper/2501.16033/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 105 references that are in the map.
- First posted on arXiv 27 Jan 2025.

## Builds on (most cited first, 25 of 25)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2405.06211/) (2024)
- [Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning](https://mlsanity.com/paper/1712.03141/) (2017)
- [In-Context Impersonation Reveals Large Language Models' Strengths and Biases](https://mlsanity.com/paper/2305.14930/) (2023)
- [Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment](https://mlsanity.com/paper/2402.14016/) (2024)
- [The Metacognitive Demands and Opportunities of Generative AI](https://mlsanity.com/paper/2312.10893/) (2023)
- ["I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust](https://mlsanity.com/paper/2405.00623/) (2024)
- [Polisis: Automated Analysis and Presentation of Privacy Policies Using Deep Learning](https://mlsanity.com/paper/1802.02561/) (2018)
- [CLEAR: Towards Contextual LLM-Empowered Privacy Policy Analysis and Risk Generation for Large Language Model Applications](https://mlsanity.com/paper/2410.13387/) (2024)
- [Deepfakes, Phrenology, Surveillance, and More! A Taxonomy of AI Privacy Risks](https://mlsanity.com/paper/2310.07879/) (2023)
- [Large Language Models: A New Approach for Privacy Policy Analysis at Scale](https://mlsanity.com/paper/2405.20900/) (2024)
- [Empowering Users in Digital Privacy Management through Interactive LLM-Based Agents](https://mlsanity.com/paper/2410.11906/) (2024)
- [AI-enabled Automation for Completeness Checking of Privacy Policies](https://mlsanity.com/paper/2106.05688/) (2021)
- [GenAIPABench: A Benchmark for Generative AI-based Privacy Assistants](https://mlsanity.com/paper/2309.05138/) (2023)
- [TILT: A GDPR-Aligned Transparency Information Language and Toolkit for Practical Privacy Engineering](https://mlsanity.com/paper/2012.10431/) (2020)
- [Truth in Motion: The Unprecedented Risks and Opportunities of Extended Reality Motion Data](https://mlsanity.com/paper/2306.06459/) (2023)
- [Feedback Exchange and Online Affinity: A Case Study of Online Fanfiction Writers](https://mlsanity.com/paper/2209.12810/) (2022)
- [Exploring the Online Micro-targeting Practices of Small, Medium, and Large Businesses](https://mlsanity.com/paper/2207.09286/) (2022)
- [Enabling Versatile Privacy Interfaces Using Machine-Readable Transparency Information](https://mlsanity.com/paper/2302.10991/) (2023)
- [A Human-in-the-Loop Approach for Information Extraction from Privacy Policies under Data Scarcity](https://mlsanity.com/paper/2305.15006/) (2023)
- [Identifying Privacy Personas](https://mlsanity.com/paper/2410.14023/) (2024)
- [Bolder is Better: Raising User Awareness through Salient and Concise Privacy Notices](https://mlsanity.com/paper/2101.08021/) (2021)
- [Legally Binding but Unfair? Towards Assessing Fairness of Privacy Policies](https://mlsanity.com/paper/2403.08115/) (2024)

## Built on by (most cited first, 5 of 5)

- [Position: Privacy Is Not Just Memorization!](https://mlsanity.com/paper/2510.01645/) (2025)
- [A Comprehensive Review on Harnessing Large Language Models to Overcome Recommender System Challenges](https://mlsanity.com/paper/2507.21117/) (2025)
- [Understanding U.S. Users' Security and Privacy Transparency Needs for Consumer-Facing Generative AI](https://mlsanity.com/paper/2604.17270/) (2026)
- [The Privacy Guardian Agent: Towards Trustworthy AI Privacy Agents](https://mlsanity.com/paper/2604.21455/) (2026)
- [GuardianAgent: Policy-Conditioned Risk-Adaptive Anonymization with Verified Adversarial Escalation](https://mlsanity.com/paper/2608.29251/) (2026)

## Abstract (arXiv)

Understanding and engaging with privacy policies is crucial for online privacy, yet these documents remain notoriously complex and difficult to navigate. We present PRISMe, an interactive browser extension that combines LLM-based policy assessment with a dashboard and customizable chat interface, enabling users to skim quick overviews or explore policy details in depth while browsing. We conduct a user study (N=22) with participants of diverse privacy knowledge to investigate how users interpret the tool's explanations and how it shapes their engagement with privacy policies, identifying distinct interaction patterns. Participants valued the clear overviews and conversational depth, but flagged some issues, particularly adversarial robustness and hallucination risks. Thus, we investigate how a retrieval-augmented generation (RAG) approach can alleviate issues by re-running the chat queries from the study. Our findings surface design challenges as well as technical trade-offs, contributing actionable insights for developing future user-centered, trustworthy privacy policy analysis tools.