mlsanityData to 7 Oct 2026

Oracle-Robust Online Alignment for Large Language Models

arXiv 2602.20457 ↗First posted 24 Feb 2026cs.LG stat.MLcited by 1

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

We study online alignment of large language models under misspecified preference feedback, where the observed preference oracle deviates from an ideal but unknown ground-truth oracle. The online LLM alignment problem is a bi-level reinforcement problem due to the coupling between data collection and policy updates. Recently, the problem has been reduced to tractable single-level objective in the SAIL (Self-Improving Efficient Online Alignment) framework. In this paper, we introduce a pointwise oracle uncertainty set in this problem and formulate an oracle-robust online alignment objective as a worst-case optimization problem. For log-linear policies, we show that this robust objective admits an exact closed-form decomposition into the original loss function plus an explicit sensitivity penalty. We develop projected stochastic composite updates for the resulting weakly convex objective and prove widetildeO(varepsilon-2) oracle complexity for reaching approximate stationarity.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 46 references that are in the map. First posted on arXiv 24 Feb 2026.

Builds on

33 of 46 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimization4 Feb
  2. 2025Online Preference Alignment for Language Models via Count-based Exploration22 Jan
  3. 2024Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift26 Jul
  4. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization10 Jul
  5. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Models21 Jun
  6. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignment29 May
  7. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 26

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2015Trust Region Policy Optimizationcited by 2,503
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  8. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  9. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  10. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  11. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 403
  12. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
14 more
  1. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 334
  2. 2015Data-driven Distributionally Robust Optimization Using the Wasserstein Metric: Performance Guarantees and Tractable Reformulationscited by 285
  3. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
  4. 2019Wasserstein Distributionally Robust Optimization: Theory and Applications in Machine Learningcited by 169
  5. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  6. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
  7. 2023On Penalty-based Bilevel Gradient Descent Methodcited by 87
  8. 2024Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHFcited by 41
  9. 2023On Finding Small Hyper-Gradients in Bilevel Optimization: Hardness Results and Improved Analysiscited by 39
  10. 2023PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedbackcited by 38
  11. 2023Goodhart's Law in Reinforcement Learningcited by 37
  12. 2018Stochastic subgradient method converges at the rate O(k-1/4) on weakly convex functionscited by 31
  13. 2025On The Sample Complexity Bounds In Bilevel Reinforcement Learningcited by 10
  14. 2021Convergence and Optimality of Policy Gradient Methods in Weakly Smooth Settingscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Distributionally Robust Listwise Preference Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Sample-Efficient Alignment for LLMs
  2. 2025ComPO: Preference Alignment via Comparison Oracles
  3. 2026A Zeroth-Order Paradigm for LLM Preference Alignment
  4. 2025Alignment of large language models with constrained learning
  5. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Time
  6. 2025Towards Reliable, Uncertainty-Aware Alignment
  7. 2026Asymptotic Universal Alignment: A New Alignment Framework via Test-Time Scaling
  8. 2025DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models