# A Human-in-the-Loop Approach for Information Extraction from Privacy Policies under Data Scarcity

arXiv: https://arxiv.org/abs/2305.15006 · HTML: https://mlsanity.com/paper/2305.15006/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 32 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 11 of 11)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [Polisis: Automated Analysis and Presentation of Privacy Policies Using Deep Learning](https://mlsanity.com/paper/1802.02561/) (2018)
- [AI-enabled Automation for Completeness Checking of Privacy Policies](https://mlsanity.com/paper/2106.05688/) (2021)
- [TILT: A GDPR-Aligned Transparency Information Language and Toolkit for Practical Privacy Engineering](https://mlsanity.com/paper/2012.10431/) (2020)
- [Machine Understandable Policies and GDPR Compliance Checking](https://mlsanity.com/paper/2001.08930/) (2020)
- [Enabling Versatile Privacy Interfaces Using Machine-Readable Transparency Information](https://mlsanity.com/paper/2302.10991/) (2023)

## Built on by (most cited first, 2 of 2)

- [Helping Johnny Make Sense of Privacy Policies with LLMs](https://mlsanity.com/paper/2501.16033/) (2025)
- [A Multi-solution Study on GDPR AI-enabled Completeness Checking of DPAs](https://mlsanity.com/paper/2311.13881/) (2023)

## Abstract (arXiv)

Machine-readable representations of privacy policies are door openers for a broad variety of novel privacy-enhancing and, in particular, transparency-enhancing technologies (TETs). In order to generate such representations, transparency information needs to be extracted from written privacy policies. However, respective manual annotation and extraction processes are laborious and require expert knowledge. Approaches for fully automated annotation, in turn, have so far not succeeded due to overly high error rates in the specific domain of privacy policies. In the end, a lack of properly annotated privacy policies and respective machine-readable representations persists and enduringly hinders the development and establishment of novel technical approaches fostering policy perception and data subject informedness. In this work, we present a prototype system for a `Human-in-the-Loop' approach to privacy policy annotation that integrates ML-generated suggestions and ultimately human annotation decisions. We propose an ML-based suggestion system specifically tailored to the constraint of data scarcity prevalent in the domain of privacy policy annotation. On this basis, we provide meaningful predictions to users thereby streamlining the annotation process. Additionally, we also evaluate our approach through a prototypical implementation to show that our ML-based extraction approach provides superior performance over other recently used extraction models for legal documents.