mlsanityData to 14 Oct 2026

Towards User-level Private Reinforcement Learning with Human Feedback

arXiv 2502.17515 ↗First posted 22 Feb 2025cs.LG cs.AIcited by 14

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

also in 1 more
Abstract (arXiv)

Reinforcement Learning with Human Feedback (RLHF) has emerged as an influential technique, enabling the alignment of large language models (LLMs) with human preferences. Despite the promising potential of RLHF, how to protect user preference privacy has become a crucial issue. Most previous work has focused on using differential privacy (DP) to protect the privacy of individual data. However, they have concentrated primarily on item-level privacy protection and have unsatisfactory performance for user-level privacy, which is more common in RLHF. This study proposes a novel framework, AUP-RLHF, which integrates user-level label DP into RLHF. We first show that the classical random response algorithm, which achieves an acceptable performance in item-level privacy, leads to suboptimal utility when in the user-level settings. We then establish a lower bound for the user-level label DP-RLHF and develop the AUP-RLHF algorithm, which guarantees (varepsilon, δ) user-level privacy and achieves an improved estimation error. Experimental results show that AUP-RLHF outperforms existing baseline methods in sentiment generation and summarization tasks, achieving a better privacy-utility trade-off.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 56 references that are in the map. First posted on arXiv 22 Feb 2025.

Builds on

33 of 56 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Privately Aligning Language Models with Reinforcement Learning25 Oct

Other references · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2016Deep Learning with Differential Privacycited by 2,728
  5. 2017Differentially Private Federated Learning: A Client Level Perspectivecited by 459
  6. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  7. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 333
  8. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  9. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 247
  10. 2023Using an LLM to Help With Code Understandingcited by 228
  11. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 136
  12. 2018Differentially Private Empirical Risk Minimization Revisited: Faster and More Generalcited by 128
20 more
  1. 2021Deep Learning with Label Differential Privacycited by 117
  2. 2023The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Developmentcited by 96
  3. 2024On Protecting the Data Privacy of Large Language Models (LLMs): A Surveycited by 96
  4. 2024CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needscited by 84
  5. 2023Leveraging Large Language Models for Sequential Recommendationcited by 78
  6. 2021Learning with User-Level Privacycited by 72
  7. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimismcited by 65
  8. 2015Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependencecited by 61
  9. 2023Precedent-Enhanced Legal Judgment Prediction with LLM and Domain-Model Collaborationcited by 52
  10. 2021High Dimensional Differentially Private Stochastic Optimization with Heavy-tailed Datacited by 42
  11. 2024Locate-then-edit for Multi-hop Factual Recall under Knowledge Editingcited by 37
  12. 2024Multi-hop Question Answering under Temporal Knowledge Editingcited by 36
  13. 2024A Continued Pretrained LLM Approach for Automatic Medical Note Generationcited by 30
  14. 2024Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuningcited by 24
  15. 2024Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domaincited by 15
  16. 2023User-Level Differential Privacy With Few Examples Per Usercited by 12
  17. 2023Differentially Private Reward Estimation with Preference Feedbackcited by 12
  18. 2023User-level Differentially Private Stochastic Convex Optimization: Efficient Algorithms with Optimal Ratescited by 12
  19. 2024A Huber Loss Minimization Approach to Mean Estimation under User-level Differential Privacycited by 11
  20. 2024Private Language Models via Truncated Laplacian Mechanismcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 8.

14 papers.

Sort:

By area:

  1. 2025When Truth Is Overridden: Uncovering the Internal Origins of Sycophancy in Large Language Modelscited by 50
  2. 2025Curriculum-RLAIF: Curriculum Alignment with Reinforcement Learning from AI Feedbackcited by 32
  3. 2025Understanding How Value Neurons Shape the Generation of Specified Values in LLMscited by 13
  4. 2026Predicting LLM Output Length via Entropy-Guided Representationscited by 9
  5. 2025Towards Reasoning-Preserving Unlearning in Multimodal Large Language Modelscited by 4
  6. 2025KL-regularization Itself is Differentially Private in Bandits and RLHFcited by 3
  7. 2026Concept-Based Dictionary Learning for Inference-Time Safety in Vision Language Action Modelscited by 3
  8. 2025Attributing Data for Sharpness-Aware Minimizationcited by 2
  9. 2026CoLA: A Choice Leakage Attack Framework to Expose Privacy Risks in Subset Trainingcited by 2
  10. 2026Understanding the Impact of Differentially Private Training on Memorization of Long-Tailed Datacited by 1
  11. 2026Differentially Private Non-convex Distributionally Robust Optimizationcited by 1
  12. 2026Differentially Private Preference Data Synthesis for Large Language Model Alignmentcited by 1
2 more
  1. 2026Balancing Privacy, Utility, and Safety in LLM Alignment through Preference Optimizationcited by 1
  2. 2026Reward-Driven Learning under Prompt-Level Differential Privacycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Privacy-Preserving Reinforcement Learning from Human Feedback via Decoupled Reward Modeling
  2. 2024FedRLHF: A Convergence-Guaranteed Federated Framework for Privacy-Preserving and Personalized RLHF
  3. 2024Preserving Expert-Level Privacy in Offline Reinforcement Learning
  4. 2025Position Paper: Rethinking Privacy in RL for Sequential Decision-making in the Age of LLMs
  5. 2026Privacy Preserving Reinforcement Learning with One-Sided Feedback
  6. 2025Offline and Online KL-Regularized RLHF under Differential Privacy
  7. 2022Offline Reinforcement Learning with Differential Privacy
  8. 2025Differentially Private Policy Gradient