mlsanityData to 14 Oct 2026

Improved Bounds for Private and Robust Alignment

arXiv 2512.23816 ↗First posted 29 Dec 2025cs.LG cs.AIcited by 5

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

In this paper, we study the private and robust alignment of language models from a theoretical perspective by establishing upper bounds on the suboptimality gap in both offline and online settings. We consider preference labels subject to privacy constraints and/or adversarial corruption, and analyze two distinct interplays between them: privacy-first and corruption-first. For the privacy-only setting, we show that log loss with an MLE-style algorithm achieves near-optimal rates, in contrast to conventional wisdom. For the joint privacy-and-corruption setting, we first demonstrate that existing offline algorithms in fact provide stronger guarantees -- simultaneously in terms of corruption level and privacy parameters -- than previously known, which further yields improved bounds in the corruption-only regime. In addition, we also present the first set of results for private and robust online alignment. Our results are enabled by new uniform convergence guarantees for log loss and square loss under privacy and corruption, which we believe have broad applicability across learning theory and statistics.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 48 of its 61 references that are in the map. First posted on arXiv 29 Dec 2025.

Builds on

48 of 61 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025SquarechiPO: Differentially Private and Robust chi2-Preference Optimization in Offline Direct Alignment27 May
  2. 2025A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO21 May
  3. 2024Exposing Privacy Gaps: Membership Inference Attack on Preference Data for LLM Alignment8 Jul

Other references · 45

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  5. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  6. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  7. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  8. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 247
  9. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 233
  10. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  11. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 135
  12. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
33 more
  1. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  2. 2022The Role of Coverage in Online Reinforcement Learningcited by 83
  3. 2018Differentially Private Contextual Linear Banditscited by 82
  4. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 81
  5. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  6. 2024Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learningcited by 74
  7. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  8. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  9. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  10. 2021Cautiously Optimistic Policy Optimization and Exploration with Linear Function Approximationcited by 53
  11. 2020Private Reinforcement Learning with PAC and Regret Guaranteescited by 51
  12. 2023The History and Risks of Reinforcement Learning and Human Feedbackcited by 42
  13. 2019An Optimal Private Stochastic-MAB Algorithm Based on an Optimal Private Stopping Rulecited by 37
  14. 2020Multi-Armed Bandits with Local Differential Privacycited by 37
  15. 2023Foundations of Reinforcement Learning and Interactive Decision Makingcited by 33
  16. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 33
  17. 2019Manipulation Attacks in Local Differential Privacycited by 28
  18. 2022When Privacy Meets Partial Information: A Refined Analysis of Differentially Private Banditscited by 28
  19. 2022Shuffle Private Linear Contextual Banditscited by 27
  20. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learningcited by 25
  21. 2021Differentially Private Regret Minimization in Episodic Markov Decision Processescited by 23
  22. 2022Differentially Private Reinforcement Learning with Linear Function Approximationcited by 22
  23. 2025Logarithmic Regret for Online KL-Regularized Reinforcement Learningcited by 22
  24. 2022Near-Optimal Differentially Private Reinforcement Learningcited by 16
  25. 2022Distributed Differential Privacy in Multi-Armed Banditscited by 14
  26. 2023On Differentially Private Federated Linear Contextual Banditscited by 14
  27. 2023Differentially Private Reward Estimation with Preference Feedbackcited by 12
  28. 2021Differentially Private Exploration in Reinforcement Learning with Linear Representationcited by 11
  29. 2025Towards a Sharp Analysis of Offline Policy Learning for f-Divergence-Regularized Contextual Banditscited by 10
  30. 2025Outcome-Based Online Reinforcement Learning: Algorithms and Fundamental Limitscited by 7
  31. 2023On Private and Robust Banditscited by 6
  32. 2024Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Explorationcited by 6
  33. 2025On the Sample Complexity of Differentially Private Policy Optimizationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Near-Optimal Regret for KL-Regularized Multi-Armed Banditscited by 5
  2. 2026On the Optimal Sample Complexity of Offline Multi-Armed Bandits with KL Regularizationcited by 4
  3. 2026Private Direct Preference Optimization for LLM Alignmentcited by 0
  4. 2026WorldMark: A Plug-and-Play World Knowledge Interface for Cross-Host Language Model Watermarkingcited by 0
  5. 2026Overtaught, Underlearned: Learner-Centered Distillation from Multi-Agent Discussioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Improved Algorithms for Differentially Private Language Model Alignment
  2. 2025PROPS: Progressively Private Self-alignment of Large Language Models
  3. 2023Privately Aligning Language Models with Reinforcement Learning
  4. 2026Oracle-Robust Online Alignment for Large Language Models
  5. 2026On the Limits of Support-Preserving Alignment and Bounded Filtering
  6. 2026Personalized Alignment Revisited: The Necessity and Sufficiency of User Diversity
  7. 2026Differentially Private Preference Data Synthesis for Large Language Model Alignment
  8. 2026Privacy Without Regret: Differentially Private Inference-Time Alignment