mlsanityData to 7 Oct 2026

Online Preference Alignment for Language Models via Count-based Exploration

arXiv 2501.12735 ↗First posted 22 Jan 2025cs.LGcited by 24

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) has shown great potential in fine-tuning Large Language Models (LLMs) to align with human preferences. Existing methods perform preference alignment from a fixed dataset, which can be limited in data coverage, and the resulting reward model is hard to generalize in out-of-distribution responses. Thus, online RLHF is more desirable to empower the LLM to explore outside the support of the initial dataset by iteratively collecting the prompt-response pairs. In this paper, we study the fundamental problem in online RLHF, i.e. how to explore for LLM. We give a theoretical motivation in linear reward assumption to show that an optimistic reward with an upper confidence bound (UCB) term leads to a provably efficient RLHF policy. Then, we reformulate our objective to direct preference optimization with an exploration term, where the UCB-term can be converted to a count-based exploration bonus. We further propose a practical algorithm, named Count-based Online Preference Optimization (COPO), which leverages a simple coin-flip counting module to estimate the pseudo-count of a prompt-response pair in previously collected data. COPO encourages LLMs to balance exploration and preference optimization in an iterative manner, which enlarges the exploration space and the entire data coverage of iterative LLM policies. We conduct online RLHF experiments on Zephyr and Llama-3 models. The results on instruction-following and standard academic benchmarks show that COPO significantly increases performance.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 76 references that are in the map. First posted on arXiv 22 Jan 2025.

Builds on

39 of 76 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignment29 May
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  3. 2024D2PO: Discriminator-Guided DPO with Response Evaluation Models2 May
  4. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  5. 2024Aligning Large Language Models by On-Policy Self-Judgment17 Feb
  6. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  7. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  8. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb
  9. 2023Zephyr: Direct Distillation of LM Alignment25 Oct

Other references · 30

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  11. 2018Exploration by Random Network Distillationcited by 966
  12. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
18 more
  1. 2024Yi: Open Foundation Models by 01.AIcited by 706
  2. 2024Self-Rewarding Language Modelscited by 654
  3. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  4. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  5. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  6. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  7. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  8. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  9. 2023Nash Learning from Human Feedbackcited by 220
  10. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  11. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  12. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  13. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 103
  14. 2023EQ-Bench: An Emotional Intelligence Benchmark for Large Language Modelscited by 75
  15. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  16. 2024Efficient Exploration for LLMscited by 44
  17. 2024Regularized Conditional Diffusion Model for Multi-Task Preference Alignmentcited by 9
  18. 2024Preference Aligned Diffusion Planner for Quadrupedal Locomotion Controlcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 11.

24 papers.

Sort:

By area:

  1. 2025Outcome-based Exploration for LLM Reasoningcited by 79
  2. 2025A Survey of Direct Preference Optimizationcited by 35
  3. 2025CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Modelscited by 28
  4. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regressioncited by 23
  5. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Trainingcited by 16
  6. 2025CTRLS: Chain-of-Thought Reasoning via Latent State-Transitioncited by 13
  7. 2025The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic Viewcited by 13
  8. 2025Risk-Sensitive RL for Alleviating Exploration Dilemmas in Large Language Modelscited by 12
  9. 2025VLP: Vision-Language Preference Learning for Embodied Manipulationcited by 8
  10. 2025Information-Theoretic Reward Decomposition for Generalizable RLHFcited by 5
  11. 2026Towards a Theoretical Understanding to the Generalization of RLHFcited by 5
  12. 2026Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learningcited by 5
12 more
  1. 2025Sample-Efficient Reinforcement Learning from Human Feedback via Information-Directed Samplingcited by 4
  2. 2025SDA: Steering-Driven Distribution Alignment for Open LLMs without Fine-Tuningcited by 2
  3. 2026On Advantage Estimates for Max@K Policy Gradientscited by 2
  4. 2026A Component-Based Survey of Interactions between Large Language Models and Multi-Armed Banditscited by 1
  5. 2026Oracle-Robust Online Alignment for Large Language Modelscited by 1
  6. 2026Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelinescited by 1
  7. 2025Multi-Objective Reward and Preference Optimization: Theory and Algorithmscited by 0
  8. 2026Real-Time Trend Prediction via Continually-Aligned LLM Query Generationcited by 0
  9. 2026Pref-CTRL: Preference Driven LLM Alignment using Representation Editingcited by 0
  10. 2026Boosting Reinforcement Learning with Verifiable Rewards via Randomly Selected Few-Shot Guidancecited by 0
  11. 2026DRIFT: Decoupled Rollouts and Importance-Weighted Fine-Tuning for Efficient Multi-Turn Optimizationcited by 0
  12. 2026N-GRPO: Embedding-Level Neighbor Mixing for Enhanced Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Avoiding mathbfexp(Rmax) scaling in RLHF through Preference-based Exploration
  2. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Models
  3. 2024Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration
  4. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  5. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization
  6. 2024OPTune: Efficient Online Preference Tuning
  7. 2023Sample Efficient Preference Alignment in LLMs via Active Exploration
  8. 2024Investigating on RLHF methodology