mlsanityData to 14 Oct 2026

KL-regularization Itself is Differentially Private in Bandits and RLHF

arXiv 2505.18407 ↗First posted 23 May 2025cs.LG cs.AIcited by 3

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Differential Privacy (DP) provides a rigorous framework for privacy, ensuring the outputs of data-driven algorithms remain statistically indistinguishable across datasets that differ in a single entry. While guaranteeing DP generally requires explicitly injecting noise either to the algorithm itself or to its outputs, the intrinsic randomness of existing algorithms presents an opportunity to achieve DP ``for free''. In this work, we explore the role of regularization in achieving DP across three different decision-making problems: multi-armed bandits, linear contextual bandits, and reinforcement learning from human feedback (RLHF), in offline data settings. We show that adding KL-regularization to the learning objective (a common approach in optimization algorithms) makes the action sampled from the resulting stochastic policy itself differentially private. This offers a new route to privacy guarantees without additional noise injection, while also preserving the inherent advantage of regularization in enhancing performance.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 47 of its 65 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

47 of 65 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHF7 Nov
  2. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  3. 2023Differentially Private Reward Estimation with Preference Feedback30 Oct
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  7. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 40

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  6. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  7. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  8. 2019A Theory of Regularized Markov Decision Processescited by 286
  9. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  10. 2014RAPPOR: Randomized Aggregatable Privacy-Preserving Ordinal Responsecited by 272
  11. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  12. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 177
28 more
  1. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
  2. 2021Deep Learning with Label Differential Privacycited by 117
  3. 2015Multi-armed Bandit Models for the Optimal Design of Clinical Trials: Benefits and Challengescited by 114
  4. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycited by 85
  5. 2018Differentially Private Contextual Linear Banditscited by 82
  6. 2016Conservative Contextual Linear Banditscited by 57
  7. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  8. 2020Locally Differentially Private (Contextual) Bandits Learningcited by 53
  9. 2020Private Reinforcement Learning with PAC and Regret Guaranteescited by 51
  10. 2018Privacy-preserving Predictioncited by 48
  11. 2023The Blessing of Heterogeneity in Federated Q-Learning: Linear Speedup and Beyondcited by 36
  12. 2016Efficient Algorithms for Adversarial Contextual Learningcited by 33
  13. 2021Generalized Linear Bandits with Local Differential Privacycited by 29
  14. 2022When Privacy Meets Partial Information: A Refined Analysis of Differentially Private Banditscited by 28
  15. 2020The Trade-Offs of Private Predictioncited by 22
  16. 2018Differential Privacy Techniques for Cyber Physical Systems: A Surveycited by 21
  17. 2022Pessimism for Offline Linear Contextual Bandits using ℓp Confidence Setscited by 18
  18. 2023Privately Aligning Language Models with Reinforcement Learningcited by 17
  19. 2023Provably Feedback-Efficient Reinforcement Learning via Active Reward Learningcited by 15
  20. 2025Towards User-level Private Reinforcement Learning with Human Feedbackcited by 14
  21. 2023Concentrated Differential Privacy for Banditscited by 10
  22. 2019PAC learning with stable and private predictionscited by 9
  23. 2018Regularized Contextual Banditscited by 4
  24. 2025Differentially Private Policy Gradientcited by 3
  25. 2021Dirichlet Mechanism for Differentially Private KL Divergence Minimizationcited by 2
  26. 2019Differential Privacy in the 2020 Decennial Census and the Implications for Available Data Productscited by 1
  27. 2020Stability Enhanced Privacy and Applications in Private Stochastic Gradient Descentcited by 1
  28. 2021Differential Privacy for Credit Risk Modelcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Offline and Online KL-Regularized RLHF under Differential Privacycited by 4
  2. 2026Privacy Without Regret: Differentially Private Inference-Time Alignmentcited by 0
  3. 2026Reward-Driven Learning under Prompt-Level Differential Privacycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022On the utility and protection of optimization with differential privacy and classic regularization techniques
  2. 2026Shuffle and Joint Differential Privacy for Generalized Linear Contextual Bandits
  3. 2025On the Sample Complexity of Differentially Private Policy Optimization
  4. 2026Privacy-Preserving Reinforcement Learning from Human Feedback via Decoupled Reward Modeling
  5. 2026Randomized Least Squares Value Iteration itself is Joint Differentially Private
  6. 2025Towards Optimal Differentially Private Regret Bounds in Linear MDPs
  7. 2026Fast Rates for Offline Contextual Bandits with Forward-KL Regularization under Single-Policy Concentrability
  8. 2022Offline Reinforcement Learning with Differential Privacy