mlsanityData to 14 Oct 2026

Privately Aligning Language Models with Reinforcement Learning

arXiv 2310.16960 ↗First posted 25 Oct 2023cs.LG cs.CRcited by 17

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Positioned between pre-training and user deployment, aligning large language models (LLMs) through reinforcement learning (RL) has emerged as a prevailing strategy for training instruction following-models such as ChatGPT. In this work, we initiate the study of privacy-preserving alignment of LLMs through Differential Privacy (DP) in conjunction with RL. Following the influential work of Ziegler et al. (2020), we study two dominant paradigms: (i) alignment via RL without human in the loop (e.g., positive review generation) and (ii) alignment via RL from human feedback (RLHF) (e.g., summarization in a human-preferred way). We give a new DP framework to achieve alignment via RL, and prove its correctness. Our experimental results validate the effectiveness of our approach, offering competitive utility while ensuring strong privacy protections.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 29 of its 43 references that are in the map. First posted on arXiv 25 Oct 2023.

Builds on

29 of 43 references.

References · 29

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2016Deep Learning with Differential Privacycited by 2,728
  8. 2020Extracting Training Data from Large Language Modelscited by 2,189
  9. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  10. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  11. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  12. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
17 more
  1. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  2. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  3. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 258
  4. 2018Improving the Gaussian Mechanism for Differential Privacy: Analytical Calibration and Optimal Denoisingcited by 218
  5. 2019SemEval-2017 Task 4: Sentiment Analysis in Twittercited by 210
  6. 2021Numerical Composition of Differential Privacycited by 180
  7. 2022Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipecited by 108
  8. 2023Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Modelscited by 83
  9. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimismcited by 78
  10. 2017Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedbackcited by 72
  11. 2018Can Neural Machine Translation be Improved with User Feedback?cited by 72
  12. 2022Differentially Private Language Models for Secure Data Sharingcited by 68
  13. 2023SoK: Privacy-Preserving Data Synthesiscited by 38
  14. 2019Privacy-preserving Q-Learning with Functional Noise in Continuous State Spacescited by 30
  15. 2022Offline Reinforcement Learning with Differential Privacycited by 20
  16. 2016Differentially Private Policy Evaluationcited by 19
  17. 2022Privacy Leakage in Text Classification: A Data Extraction Approachcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 8; 2026: 7.

17 papers.

Sort:

By area:

  1. 2024Privacy-Preserving Instructions for Aligning Large Language Modelscited by 35
  2. 2025Towards User-level Private Reinforcement Learning with Human Feedbackcited by 14
  3. 2025Empirical Privacy Variancecited by 3
  4. 2025KL-regularization Itself is Differentially Private in Bandits and RLHFcited by 3
  5. 2025ACTG-ARL: Differentially Private Conditional Text Generation with RL-Boosted Controlcited by 3
  6. 2025On the Sample Complexity of Differentially Private Policy Optimizationcited by 3
  7. 2026Privacy-Preserving Reinforcement Learning from Human Feedback via Decoupled Reward Modelingcited by 3
  8. 2025Improved Algorithms for Differentially Private Language Model Alignmentcited by 2
  9. 2024Differentially Private Reinforcement Learning with Self-Playcited by 1
  10. 2025PROPS: Progressively Private Self-alignment of Large Language Modelscited by 1
  11. 2025Autonomy Reshapes How Personalization Affects Privacy Concerns and Trust in LLM Agentscited by 1
  12. 2026Autoregressive Direct Preference Optimizationcited by 1
5 more
  1. 2026Differentially Private Preference Data Synthesis for Large Language Model Alignmentcited by 1
  2. 2026Balancing Privacy, Utility, and Safety in LLM Alignment through Preference Optimizationcited by 1
  3. 2026Protecting Private Code in IDE Autocomplete using Differential Privacycited by 0
  4. 2026PrivMedChat: End-to-End Differentially Private RLHF for Medical Dialogue Systemscited by 0
  5. 2026S2T-RLHF: Hierarchical Credit Assignment for Stable Preference-Based RLHFcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Differentially Private Steering for Large Language Model Alignment
  2. 2025SquarechiPO: Differentially Private and Robust chi2-Preference Optimization in Offline Direct Alignment
  3. 2025Improved Bounds for Private and Robust Alignment
  4. 2025Efficient Differentially Private Fine-Tuning of LLMs via Reinforcement Learning
  5. 2025Privacy-R1: Privacy-Aware Multi-LLM Agent Collaboration via Reinforcement Learning
  6. 2025Offline and Online KL-Regularized RLHF under Differential Privacy
  7. 2025RL-Finetuned LLMs for Privacy-Preserving Synthetic Rewriting
  8. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment