# Generating Privacy Stories From Software Documentation

arXiv: https://arxiv.org/abs/2506.23014 · HTML: https://mlsanity.com/paper/2506.23014/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 35 of its 67 references that are in the map.
- First posted on arXiv 28 Jun 2025.

## Builds on (most cited first, 30 of 35)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Chain-of-Verification Reduces Hallucination in Large Language Models](https://mlsanity.com/paper/2309.11495/) (2023)
- [Large Language Models for Data Annotation and Synthesis: A Survey](https://mlsanity.com/paper/2402.13446/) (2024)
- [HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detection](https://mlsanity.com/paper/2409.17504/) (2024)
- [Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agents](https://mlsanity.com/paper/2311.11797/) (2023)
- [A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chains](https://mlsanity.com/paper/2402.00559/) (2024)
- [Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Models](https://mlsanity.com/paper/2410.07176/) (2024)
- [Beyond Chain-of-Thought: A Survey of Chain-of-X Paradigms for LLMs](https://mlsanity.com/paper/2404.15676/) (2024)
- [Thinking LLMs: General Instruction Following with Thought Generation](https://mlsanity.com/paper/2410.10630/) (2024)
- [Advancing Requirements Engineering through Generative AI: Assessing the Role of LLMs](https://mlsanity.com/paper/2310.13976/) (2023)
- [Investigating ChatGPT's Potential to Assist in Requirements Elicitation Processes](https://mlsanity.com/paper/2307.07381/) (2023)
- [A Comparative Study on Annotation Quality of Crowdsourcing and LLM via Label Aggregation](https://mlsanity.com/paper/2401.09760/) (2024)
- [Using LLMs in Software Requirements Specifications: An Empirical Evaluation](https://mlsanity.com/paper/2404.17842/) (2024)
- [PolicyGPT: Automated Analysis of Privacy Policies with Large Language Models](https://mlsanity.com/paper/2309.10238/) (2023)
- [Leveraging LLMs for the Quality Assurance of Software Requirements](https://mlsanity.com/paper/2408.10886/) (2024)
- [Large Language Models: A New Approach for Privacy Policy Analysis at Scale](https://mlsanity.com/paper/2405.20900/) (2024)
- [Foundations of Large Language Models](https://mlsanity.com/paper/2501.09223/) (2025)
- [LLM-based agents for automating the enhancement of user story quality: An early report](https://mlsanity.com/paper/2403.09442/) (2024)
- [HoT: Highlighted Chain of Thought for Referencing Supporting Facts from Inputs](https://mlsanity.com/paper/2503.02003/) (2025)
- [Unpacking Privacy Labels: A Measurement and Developer Perspective on Google's Data Safety Section](https://mlsanity.com/paper/2306.08111/) (2023)
- [ATLAS: Automatically Detecting Discrepancies Between Privacy Policies and Privacy Labels](https://mlsanity.com/paper/2306.09247/) (2023)
- [Requirements Satisfiability with In-Context Learning](https://mlsanity.com/paper/2404.12576/) (2024)

## Abstract (arXiv)

Research shows that analysts and developers consider privacy as a security concept or as an afterthought, which may lead to non-compliance and violation of users' privacy. Most current approaches, however, focus on extracting legal requirements from the regulations and evaluating the compliance of software and processes with them. In this paper, we develop a novel approach based on chain-of-thought prompting (CoT), in-context-learning (ICL), and Large Language Models (LLMs) to extract privacy behaviors from various software documents prior to and during software development, and then generate privacy requirements in the format of user stories. Our results show that most commonly used LLMs, such as GPT-4o and Llama 3, can identify privacy behaviors and generate privacy user stories with F1 scores exceeding 0.8. We also show that the performance of these models could be improved through parameter-tuning. Our findings provide insight into using and optimizing LLMs for generating privacy requirements given software documents created prior to or throughout the software development lifecycle.