mlsanityData to 14 Oct 2026

Offline and Online KL-Regularized RLHF under Differential Privacy

arXiv 2510.13512 ↗First posted 15 Oct 2025cs.LG cs.AIcited by 4

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

In this paper, we study the offline and online settings of reinforcement learning from human feedback (RLHF) with KL-regularization -- a widely used objective function in large language model alignment -- under the ε local differential privacy (ε-LDP) model on the label of the human preference. In the offline setting, we design an algorithm based on the principle of pessimism and derive a new suboptimality gap of tildeO(1/[(eε-1)2 n]) on the KL-regularized objective under single-policy concentrability. We also prove its optimality by providing a matching lower bound where n is the sample size. In the online setting, we are the first one to theoretically investigate the problem of KL-regularized RLHF with LDP. We design an optimism-based algorithm and derive a logarithmic regret bound of O(dFlog (NF· T) /(eε-1)2 ), where T is the total time step, NF is cardinality of the reward function space F and dF is a variant of eluder dimension for RLHF. As a by-product of our analysis, our results also imply the first analysis for online KL-regularized RLHF without privacy. We implement our algorithm in the offline setting to verify our theoretical results and release our open source code at: https://github.com/rushil-thareja/PPKL-RLHF-Official.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 41 references that are in the map. First posted on arXiv 15 Oct 2025.

Builds on

33 of 41 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025KL-regularization Itself is Differentially Private in Bandits and RLHF23 May
  2. 2025Is a Good Foundation Necessary for Efficient Reinforcement Learning? The Computational Role of the Base Model in Exploration10 Mar
  3. 2025Logarithmic Regret for Online KL-Regularized Reinforcement Learning11 Feb
  4. 2025KL-Regularized RLHF with Multiple Reference Models: Exact Solutions and Sample Complexity3 Feb
  5. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHF7 Nov
  6. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF31 May
  7. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF29 May
  8. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Model11 Feb
  9. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  10. 2023Differentially Private Reward Estimation with Preference Feedback30 Oct
  11. 2023Making RL with Preference-based Feedback Efficient via Randomization23 Oct
  12. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
1 more
  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 20

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  3. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  4. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  5. 2021The Statistical Complexity of Interactive Decision Makingcited by 218
  6. 2021Deep Learning with Label Differential Privacycited by 117
  7. 2014Model-based Reinforcement Learning and the Eluder Dimensioncited by 116
  8. 2022The Role of Coverage in Online Reinforcement Learningcited by 83
  9. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 81
  10. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  11. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  12. 2022GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyondcited by 46
8 more
  1. 2022Corruption-Robust Algorithms with Uncertainty Weighting for Nonlinear Contextual Bandits and Markov Decision Processescited by 33
  2. 2025Optimistically Optimistic Exploration for Provably Efficient Infinite-Horizon Reinforcement and Imitation Learningcited by 12
  3. 2025Towards a Sharp Analysis of Offline Policy Learning for f-Divergence-Regularized Contextual Banditscited by 10
  4. 2023Optimistic Planning by Regularized Dynamic Programmingcited by 8
  5. 2025A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPOcited by 8
  6. 2025Outcome-Based Online Reinforcement Learning: Algorithms and Fundamental Limitscited by 7
  7. 2025Be Careful When Fine-tuning On Open-Source LLMs: Your Fine-tuning Data Could Be Secretly Stolen!cited by 5
  8. 2025Do Large Language Models (Really) Need Statistical Foundations?cited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Near-Optimal Regret for KL-Regularized Multi-Armed Banditscited by 5
  2. 2026On the Optimal Sample Complexity of Offline Multi-Armed Bandits with KL Regularizationcited by 4
  3. 2026When Greedy Sampling Explores: KL-Regularized Contextual Bandits without Eluder-Dimension Dependencecited by 1
  4. 2026Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectivescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026f-Divergence Regularized RLHF: Two Tales of Sampling and Unified Analyses
  2. 2026Unifying Stable Optimization and Reference Regularization in RLHF
  3. 2025Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimization
  4. 2025Towards User-level Private Reinforcement Learning with Human Feedback
  5. 2023Privately Aligning Language Models with Reinforcement Learning
  6. 2024RLHF Workflow: From Reward Modeling to Online RLHF
  7. 2025Thompson Sampling in Online RLHF with General Function Approximation
  8. 2026Generalisation of RLHF under Reward Shift and Clipped KL Regularisation