# PolicyQA: A Reading Comprehension Dataset for Privacy Policies

arXiv: https://arxiv.org/abs/2010.02557 · HTML: https://mlsanity.com/paper/2010.02557/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2020.

## Built on by (most cited first, 30 of 34)

- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [A Survey on Data Augmentation in Large Model Era](https://mlsanity.com/paper/2401.15422/) (2024)
- [Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2502.06872/) (2025)
- [ConditionalQA: A Complex Reading Comprehension Dataset with Conditional Answers](https://mlsanity.com/paper/2110.06884/) (2021)
- [Text Data Augmentation for Large Language Models: A Comprehensive Survey of Methods, Challenges, and Opportunities](https://mlsanity.com/paper/2501.18845/) (2025)
- [Intent Classification and Slot Filling for Privacy Policies](https://mlsanity.com/paper/2101.00123/) (2021)
- [VNHSGE: VietNamese High School Graduation Examination Dataset for Large Language Models](https://mlsanity.com/paper/2305.12199/) (2023)
- [Bringing order into the realm of Transformer-based language models for artificial intelligence and law](https://mlsanity.com/paper/2308.05502/) (2023)
- [Retrieval-Augmented Data Augmentation for Low-Resource Domain Tasks](https://mlsanity.com/paper/2402.13482/) (2024)
- [Retrieval Enhanced Data Augmentation for Question Answering on Privacy Policies](https://mlsanity.com/paper/2204.08952/) (2022)
- [Can LLMs Augment Low-Resource Reading Comprehension Datasets? Opportunities and Challenges](https://mlsanity.com/paper/2309.12426/) (2023)
- [Empowering Users in Digital Privacy Management through Interactive LLM-Based Agents](https://mlsanity.com/paper/2410.11906/) (2024)
- [Cheap and Good? Simple and Effective Data Augmentation for Low Resource Machine Reading](https://mlsanity.com/paper/2106.04134/) (2021)
- [Privacy Policy Question Answering Assistant: A Query-Guided Extractive Summarization Approach](https://mlsanity.com/paper/2109.14638/) (2021)
- [PLUE: Language Understanding Evaluation Benchmark for Privacy Policies in English](https://mlsanity.com/paper/2212.10011/) (2022)
- [GRAF: Graph Retrieval Augmented by Facts for Romanian Legal Multi-Choice Question Answering](https://mlsanity.com/paper/2412.04119/) (2024)
- [Benchmarking LLM Privacy Recognition for Social Robot Decision Making](https://mlsanity.com/paper/2507.16124/) (2025)
- [A Multi-Agent Framework for Mitigating Dialect Biases in Privacy Policy Question-Answering Systems](https://mlsanity.com/paper/2506.02998/) (2025)
- [A Longitudinal Measurement of Privacy Policy Evolution for Large Language Models](https://mlsanity.com/paper/2511.21758/) (2025)
- [Knowledge Graph Representations for LLM-Based Policy Compliance Reasoning](https://mlsanity.com/paper/2604.27713/) (2026)
- [Automated Detection and Analysis of Data Practices Using A Real-World Corpus](https://mlsanity.com/paper/2402.11006/) (2024)
- [Let's Measure the Elephant in the Room: Facilitating Personalized Automated Analysis of Privacy Policies at Scale](https://mlsanity.com/paper/2507.14214/) (2025)
- [Overcoming Language Barriers: Multilingual Analysis of the 2023 Swiss Privacy Law's Impact](https://mlsanity.com/paper/2510.05860/) (2025)
- [Exploring the State of the Art in Legal QA Systems](https://mlsanity.com/paper/2304.06623/) (2023)
- [Experts-in-the-Loop: Establishing an Effective Workflow in Crafting Privacy Q&A](https://mlsanity.com/paper/2311.11161/) (2023)
- [Demystifying Legalese: An Automated Approach for Summarizing and Analyzing Overlaps in Privacy Policies and Terms of Service](https://mlsanity.com/paper/2404.13087/) (2024)
- [An LLM-enabled semantic-centric framework to consume privacy policies](https://mlsanity.com/paper/2509.01716/) (2025)
- [Retrieval Improvements Do Not Guarantee Better Answers: A Study of RAG for AI Policy QA](https://mlsanity.com/paper/2603.24580/) (2026)
- [APPSI-139: A Parallel Corpus of English Application Privacy Policy Summarization and Interpretation](https://mlsanity.com/paper/2604.27550/) (2026)
- [SentinelRAG: Synthetic Sentinel Knowledge for RAG Database Copyright Protection](https://mlsanity.com/paper/2606.05787/) (2026)

## Abstract (arXiv)

Privacy policy documents are long and verbose. A question answering (QA) system can assist users in finding the information that is relevant and important to them. Prior studies in this domain frame the QA task as retrieving the most relevant text segment or a list of sentences from the policy document given a question. On the contrary, we argue that providing users with a short text span from policy documents reduces the burden of searching the target information from a lengthy text segment. In this paper, we present PolicyQA, a dataset that contains 25,017 reading comprehension style examples curated from an existing corpus of 115 website privacy policies. PolicyQA provides 714 human-annotated questions written for a wide range of privacy practices. We evaluate two existing neural QA models and perform rigorous analysis to reveal the advantages and challenges offered by PolicyQA.