A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO

arXiv 2505.15694 ↗First posted 21 May 2025cs.LG cs.AIcited by 8

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

In this paper, we theoretically investigate the effects of noisy labels in offline alignment, with a focus on the interplay between privacy and robustness against adversarial corruption. Specifically, under linear modeling assumptions, we present a unified analysis covering both reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) under different privacy-corruption scenarios, such as Local differential privacy-then-Corruption (LTC), where human preference labels are privatized before being corrupted by an adversary, and Corruption-then-Local differential privacy (CTL), where labels are corrupted before privacy protection. Our analysis leverages a reduction framework that reduces the offline alignment problem under linear modeling assumptions to parameter estimation in logistic regression. This framework allows us to establish an interesting separation result between LTC and CTL, demonstrating that LTC presents a greater challenge than CTL in offline alignment, even under linear models. As important by-products, our findings also advance the state-of-the-art theoretical results in offline alignment under privacy-only or corruption-only scenarios.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 48 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

33 of 48 references.

Closest ancestors · 1

Same direction, newest first

2024Exposing Privacy Gaps: Membership Inference Attack on Preference Data for LLM Alignment8 Jul

Other references · 32

2022Training language models to follow instructions with human feedbackcited by 15,971
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
2019Fine-Tuning Language Models from Human Preferencescited by 2,150
2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
2015An Introduction to Matrix Concentration Inequalitiescited by 397
2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 247
2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 135
2021Deep Learning with Label Differential Privacycited by 117
2016Minimax Optimal Procedures for Locally Private Estimationcited by 107

20 more

2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 106
2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
2024Reinforcement Learning for LLM Post-Training: A Surveycited by 92
2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learningcited by 83
2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 81
2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
2018Robust Estimation via Robust Gradient Estimationcited by 76
2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
2021Cautiously Optimistic Policy Optimization and Exploration with Linear Function Approximationcited by 53
2023The History and Risks of Reinforcement Learning and Human Feedbackcited by 42
2021Robust Policy Gradient against Strong Data Corruptioncited by 30
2019Manipulation Attacks in Local Differential Privacycited by 28
2024Robust Reinforcement Learning from Corrupted Human Feedbackcited by 25
2024Corruption Robust Offline Reinforcement Learning with Human Feedbackcited by 20
2022On robustness and local differential privacycited by 15
2020Classification Under Misspecification: Halfspaces, Generalized Linear Models, and Connections to Evolvabilitycited by 13
2023Differentially Private Reward Estimation with Preference Feedbackcited by 12
2022Trimmed Maximum Likelihood Estimation for Robust Learning in Generalized Linear Modelscited by 6
2022Robust Estimation of Discrete Distributions under Local Differential Privacycited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 5.

8 papers.

2025Improved Bounds for Private and Robust Alignmentcited by 5
2026Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Trainingcited by 5
2025Offline and Online KL-Regularized RLHF under Differential Privacycited by 4
2026Reinforcement Learning from Human Feedback: A Statistical Perspectivecited by 4
2025SquarechiPO: Differentially Private and Robust chi2-Preference Optimization in Offline Direct Alignmentcited by 3
2026A Jointly Efficient and Optimal Algorithm for Heteroskedastic Generalized Linear Bandits with Adversarial Corruptionscited by 1
2026Offline Constrained RLHF with Multiple Preference Oraclescited by 1
2026Privacy Without Regret: Differentially Private Inference-Time Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way