# Retrieval Enhanced Data Augmentation for Question Answering on Privacy Policies

arXiv: https://arxiv.org/abs/2204.08952 · HTML: https://mlsanity.com/paper/2204.08952/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 34 references that are in the map.
- First posted on arXiv 19 Apr 2022.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Efficient Natural Language Response Suggestion for Smart Reply](https://mlsanity.com/paper/1705.00652/) (2017)
- [Not Enough Data? Deep Learning to the Rescue!](https://mlsanity.com/paper/1911.03118/) (2019)
- [Data Augmentation using Pre-trained Transformer Models](https://mlsanity.com/paper/2003.02245/) (2020)
- [Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring](https://mlsanity.com/paper/1905.01969/) (2019)
- [Retrieval Augmented Code Generation and Summarization](https://mlsanity.com/paper/2108.11601/) (2021)
- [Question Answering for Privacy Policies: Combining Computational and Legal Perspectives](https://mlsanity.com/paper/1911.00841/) (2019)
- [Polisis: Automated Analysis and Presentation of Privacy Policies Using Deep Learning](https://mlsanity.com/paper/1802.02561/) (2018)
- [PolicyQA: A Reading Comprehension Dataset for Privacy Policies](https://mlsanity.com/paper/2010.02557/) (2020)
- [Intent Classification and Slot Filling for Privacy Policies](https://mlsanity.com/paper/2101.00123/) (2021)
- [Neural Retrieval for Question Answering with Cross-Attention Supervised Data Augmentation](https://mlsanity.com/paper/2009.13815/) (2020)
- [Evaluating the Values of Sources in Transfer Learning](https://mlsanity.com/paper/2104.12567/) (2021)
- [Cheap and Good? Simple and Effective Data Augmentation for Low Resource Machine Reading](https://mlsanity.com/paper/2106.04134/) (2021)
- [Privacy Policy Question Answering Assistant: A Query-Guided Extractive Summarization Approach](https://mlsanity.com/paper/2109.14638/) (2021)

## Built on by (most cited first, 10 of 10)

- [MapCoder: Multi-Agent Code Generation for Competitive Problem Solving](https://mlsanity.com/paper/2405.11403/) (2024)
- [A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations](https://mlsanity.com/paper/2407.04069/) (2024)
- [ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoning](https://mlsanity.com/paper/2403.09028/) (2024)
- [xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval](https://mlsanity.com/paper/2303.03004/) (2023)
- [CODESIM: Multi-Agent Code Generation and Problem Solving through Simulation-Driven Planning and Debugging](https://mlsanity.com/paper/2502.05664/) (2025)
- [Transcending Language Boundaries: Harnessing LLMs for Low-Resource Language Translation](https://mlsanity.com/paper/2411.11295/) (2024)
- [PLUE: Language Understanding Evaluation Benchmark for Privacy Policies in English](https://mlsanity.com/paper/2212.10011/) (2022)
- [VideoLights: Feature Refinement and Cross-Task Alignment Transformer for Joint Video Highlight Detection and Moment Retrieval](https://mlsanity.com/paper/2412.01558/) (2024)
- [Beyond Adapter Retrieval: Latent Geometry-Preserving Composition via Sparse Task Projection](https://mlsanity.com/paper/2410.09908/) (2024)
- [Experts-in-the-Loop: Establishing an Effective Workflow in Crafting Privacy Q&A](https://mlsanity.com/paper/2311.11161/) (2023)

## Abstract (arXiv)

Prior studies in privacy policies frame the question answering (QA) task as identifying the most relevant text segment or a list of sentences from a policy document given a user query. Existing labeled datasets are heavily imbalanced (only a few relevant segments), limiting the QA performance in this domain. In this paper, we develop a data augmentation framework based on ensembling retriever models that captures the relevant text segments from unlabeled policy documents and expand the positive examples in the training set. In addition, to improve the diversity and quality of the augmented data, we leverage multiple pre-trained language models (LMs) and cascade them with noise reduction filter models. Using our augmented data on the PrivacyQA benchmark, we elevate the existing baseline by a large margin (10% F1) and achieve a new state-of-the-art F1 score of 50%. Our ablation studies provide further insights into the effectiveness of our approach.