mlsanityData to 14 Oct 2026

SquarechiPO: Differentially Private and Robust chi2-Preference Optimization in Offline Direct Alignment

arXiv 2505.21395 ↗First posted 27 May 2025cs.LGcited by 3

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

In this paper, we theoretically study the offline alignment of language models with human preference feedback, under both preference label corruption and privacy protections. To this end, we propose SquarechiPO, a simple one-line change to chiPO where the standard log-loss is replaced by a new square loss over probability. Thanks to the inherent properties of this new loss, we have advanced the state-of-the-art of differentially private and robust offline direct alignment. Specifically, for the local model of label privacy, SquarechiPO is the first algorithm that attains an optimal rate based on single-policy concentrability even with general function approximations. It also gives the first result under the central model of privacy protection over both prompts (responses) and labels. On the robustness side against Huber label corruption, SquarechiPO is the first alignment method that has a meaningful theoretical guarantee under general function approximations. More importantly, SquarechiPO can address privacy protection and corruption simultaneously, where an interesting separation is observed, implying that the order of privacy and corruption matters. Furthermore, we show that SquarechiPO can also be easily extended to handle the scenario of the general preference model with state-of-the-art guarantees under corruption and privacy. Last but not least, all of our theoretical guarantees enjoy a unified analysis, building upon a new result on the generalization error bounds of least-square regression under corruption and privacy constraints, which we believe is of independent interest to the community.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 70 of its 96 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

70 of 96 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO21 May
  2. 2024Exposing Privacy Gaps: Membership Inference Attack on Preference Data for LLM Alignment8 Jul
  3. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  4. 2020Learning to summarize from human feedback2 Sep

Other references · 66

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  9. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  10. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  11. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  12. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
38 more
  1. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  2. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  3. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  4. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
  5. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 247
  6. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  7. 2023Nash Learning from Human Feedbackcited by 224
  8. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  9. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  10. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 164
  11. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  12. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 157
  13. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 144
  14. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 135
  15. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 123
  16. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  17. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  18. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  19. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  20. 2023Is RLHF More Difficult than Standard RL?cited by 96
  21. 2022The Role of Coverage in Online Reinforcement Learningcited by 83
  22. 2018Differentially Private Contextual Linear Banditscited by 82
  23. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  24. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  25. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 69
  26. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learningcited by 56
  27. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  28. 2020Locally Differentially Private (Contextual) Bandits Learningcited by 53
  29. 2020Private Reinforcement Learning with PAC and Regret Guaranteescited by 51
  30. 2022Robustness Implies Privacy in Statistical Estimationcited by 47
  31. 2024Dataset Reset Policy Optimization for RLHFcited by 41
  32. 2019An Optimal Private Stochastic-MAB Algorithm Based on an Optimal Private Stopping Rulecited by 37
  33. 2020Multi-Armed Bandits with Local Differential Privacycited by 37
  34. 2022Shuffle Private Linear Contextual Banditscited by 27
  35. 2022Privacy Induces Robustness: Information-Computation Gaps and Sparse Mean Estimationcited by 25
  36. 2021Differentially Private Regret Minimization in Episodic Markov Decision Processescited by 23
  37. 2020Local Differential Privacy for Regret Minimization in Reinforcement Learningcited by 22
  38. 2022Differentially Private Reinforcement Learning with Linear Function Approximationcited by 22

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Improved Bounds for Private and Robust Alignmentcited by 5
  2. 2026Reinforcement Learning from Human Feedback: A Statistical Perspectivecited by 4
  3. 2026Privacy Without Regret: Differentially Private Inference-Time Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Improved Algorithms for Differentially Private Language Model Alignment
  2. 2025PROPS: Progressively Private Self-alignment of Large Language Models
  3. 2023Privately Aligning Language Models with Reinforcement Learning
  4. 2026Differentially Private Preference Data Synthesis for Large Language Model Alignment
  5. 2026Private Direct Preference Optimization for LLM Alignment
  6. 2026Balancing Privacy, Utility, and Safety in LLM Alignment through Preference Optimization
  7. 2025Offline and Online KL-Regularized RLHF under Differential Privacy
  8. 2026Differential Privacy Protected Preference Alignment on Survey Derived Data