mlsanityData to 7 Oct 2026

Exposing Privacy Gaps: Membership Inference Attack on Preference Data for LLM Alignment

arXiv 2407.06443 ↗First posted 8 Jul 2024cs.AIcited by 24

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) have seen widespread adoption due to their remarkable natural language capabilities. However, when deploying them in real-world settings, it is important to align LLMs to generate texts according to acceptable human standards. Methods such as Proximal Policy Optimization (PPO) and Direct Preference Optimization (DPO) have enabled significant progress in refining LLMs using human preference data. However, the privacy concerns inherent in utilizing such preference data have yet to be adequately studied. In this paper, we investigate the vulnerability of LLMs aligned using two widely used methods - DPO and PPO - to membership inference attacks (MIAs). Our study has two main contributions: first, we theoretically motivate that DPO models are more vulnerable to MIA compared to PPO models; second, we introduce a novel reference-based attack framework specifically for analyzing preference data called PREMIA (ulinePreference data ulineMIA). Using PREMIA and existing baselines we empirically show that DPO models have a relatively heightened vulnerability towards MIA.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 29 of its 54 references that are in the map. First posted on arXiv 8 Jul 2024.

Builds on

29 of 54 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Learning to summarize from human feedback2 Sep

Other references · 28

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2016Deep Learning with Differential Privacycited by 2,682
  10. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  11. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  12. 2020Extracting Training Data from Large Language Modelscited by 2,144
16 more
  1. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  2. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  3. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  4. 2021Unsolved Problems in ML Safetycited by 291
  5. 2019White-box vs Black-box: Bayes Optimal Strategies for Membership Inferencecited by 265
  6. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  7. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  8. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 188
  9. 2024LLM Dataset Inference: Did you train on my dataset?cited by 107
  10. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  11. 2024Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Modelscited by 102
  12. 2023Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibrationcited by 90
  13. 2018Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learningcited by 75
  14. 2022RelaxLoss: Defending Membership Inference Attacks without Losing Utilitycited by 35
  15. 2023Policy Optimization in RLHF: The Impact of Out-of-preference Datacited by 35
  16. 2022On the amplification of security and privacy risks by post-hoc explanations in machine learning modelscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 10; 2026: 10.

24 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  2. 2026Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystemscited by 30
  3. 2024An overview of domain-specific foundation model: key technologies, applications and challengescited by 19
  4. 2025Membership Inference Attacks on Large-Scale Models: A Surveycited by 17
  5. 2024Detecting Training Data of Large Language Models via Expectation Maximizationcited by 13
  6. 2024Privacy-preserved LLM Cascade via CoT-enhanced Policy Learningcited by 8
  7. 2025A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPOcited by 8
  8. 2025On The Fragility of Benchmark Contamination Detection in Reasoning Modelscited by 8
  9. 2025Beyond Data Privacy: New Privacy Risks for Large Language Modelscited by 6
  10. 2025Auditing Data Membership in Reinforcement Learning With Verifiable Rewardscited by 6
  11. 2026Window-based Membership Inference Attacks Against Fine-tuned Large Language Modelscited by 5
  12. 2025SquarechiPO: Differentially Private and Robust chi2-Preference Optimization in Offline Direct Alignmentcited by 3
12 more
  1. 2025Improved Bounds for Private and Robust Alignmentcited by 3
  2. 2024How Privacy-Savvy Are Large Language Models? A Case Study on Compliance and Privacy Technical Reviewcited by 2
  3. 2026Assessing Domain-Level Susceptibility to Emergent Misalignment from Narrow Finetuningcited by 2
  4. 2026Reverse-Engineering Model Editing on Language Modelscited by 2
  5. 2026LLM-enabled Applications Require System-Level Threat Monitoringcited by 2
  6. 2025PROPS: Progressively Private Self-alignment of Large Language Modelscited by 1
  7. 2026Differentially Private Preference Data Synthesis for Large Language Model Alignmentcited by 1
  8. 2025The Hidden Cost of Modeling P(X): Vulnerability to Membership Inference Attacks in Generative Text Classifierscited by 0
  9. 2026Sensitivity-Aware Retrieval-Augmented Intent Clarificationcited by 0
  10. 2026Automated Membership Inference Attacks (AutoMIA): Discovering MIA Signal Computations using LLM Agentscited by 0
  11. 2026Private Direct Preference Optimization for LLM Alignmentcited by 0
  12. 2026Balancing Privacy, Utility, and Safety in LLM Alignment through Preference Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Improved Algorithms for Differentially Private Language Model Alignment
  2. 2026Alignment Defends LLMs from Property Inference Attacks
  3. 2026On the Privacy of LLMs: An Ablation Study
  4. 2026Are LLMs Vulnerable to Preference-Undermining Attacks (PUA)? A Factorial Analysis Methodology for Diagnosing the Trade-off between Preference Alignment and Real-World Validity
  5. 2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMs
  6. 2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
  7. 2024TODO: Enhancing LLM Alignment with Ternary Preferences
  8. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment