mlsanityData to 14 Oct 2026

General Exploratory Bonus for Optimistic Exploration in RLHF

arXiv 2510.03269 ↗First posted 27 Sep 2025cs.LG cs.AI cs.CLcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Optimistic exploration is central to improving sample efficiency in reinforcement learning with human feedback, yet existing exploratory bonus methods to incentivize exploration often fail to realize optimism. We provide a theoretical analysis showing that current formulations, under KL or α-divergence regularization, unintentionally bias exploration toward high-probability regions of the reference model, thereby reinforcing conservative behavior instead of promoting discovery of uncertain regions. To address this pitfall, we introduce the General Exploratory Bonus (GEB), a novel theoretical framework that provably satisfies the optimism principle. GEB counteracts divergence-induced bias via reference-dependent reward regulation and unifies prior heuristic bonuses as special cases, while extending naturally across the full α-divergence family. Empirically, GEB consistently outperforms baselines on alignment tasks across multiple divergence settings and large language model backbones. These results demonstrate that GEB offers both a principled and practical solution for optimistic exploration in RLHF.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 47 of its 61 references that are in the map. First posted on arXiv 27 Sep 2025.

Builds on

47 of 61 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Sample-Efficient Reinforcement Learning from Human Feedback via Information-Directed Sampling8 Feb
  2. 2025Avoiding mathbfexp(Rmax) scaling in RLHF through Preference-based Exploration2 Feb
  3. 2025Segmenting Text and Learning Their Rewards for Improved RLHF in Language Model6 Jan
  4. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown1 Oct
  5. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF31 May
  6. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF29 May
  7. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  8. 2024Active Preference Optimization for Sample Efficient RLHF16 Feb
  9. 2024Reinforcement Learning from Human Feedback with Active Queries14 Feb
  10. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
1 more
  1. 2018Scalable agent alignment via reward modeling: a research direction19 Nov

Other references · 34

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Let's Verify Step by Stepcited by 3,789
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  7. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  8. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
  9. 2021Unsolved Problems in ML Safetycited by 292
  10. 2025Reasoning with Exploration: An Entropy Perspectivecited by 281
  11. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  12. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
22 more
  1. 2024Iterative Reasoning Preference Optimizationcited by 246
  2. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  3. 2023Mitigating the Alignment Tax of RLHFcited by 183
  4. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  5. 2024Free Process Rewards without Process Labelscited by 165
  6. 2024Token-level Direct Preference Optimizationcited by 135
  7. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  8. 2024Process Reward Model with Q-Value Rankingscited by 80
  9. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  10. 2024Active Preference Learning for Large Language Modelscited by 58
  11. 2022GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyondcited by 46
  12. 2024Efficient Exploration for LLMscited by 45
  13. 2025First Return, Entropy-Eliciting Explorecited by 45
  14. 2023Language Model Alignment with Elastic Resetcited by 34
  15. 2025SDPO: Segment-Level Direct Preference Optimization for Social Agentscited by 30
  16. 2024Your Weak LLM is Secretly a Strong Teacher for Alignmentcited by 20
  17. 2024Reinforcement Learning with Token-level Feedback for Controllable Text Generationcited by 16
  18. 2024Sample-Efficient Alignment for LLMscited by 16
  19. 2025MetaMind: Modeling Human Social Thoughts with Metacognitive Multi-Agent Systemscited by 13
  20. 2024Can DPO Learn Diverse Human Values? A Theoretical Scaling Lawcited by 7
  21. 2024Challenges and Future Directions of Data-Centric AI Alignmentcited by 5
  22. 2024Seeker: Towards Exception Safety Code Generation with Intermediate Language Agents Frameworkcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026LAD: Learning Advantage Distribution for Reasoningcited by 2
  2. 2026Wasserstein Distributionally Robust Regret Optimization for Reinforcement Learning from Human Feedbackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Optimistic World Models: Efficient Exploration in Model-Based Deep Reinforcement Learning
  2. 2024Beyond Optimism: Exploration With Partially Observable Rewards
  3. 2025Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback
  4. 2024Efficient Model-Based Reinforcement Learning Through Optimistic Thompson Sampling
  5. 2026f-Divergence Regularized RLHF: Two Tales of Sampling and Unified Analyses
  6. 2025Minimax Optimal Reinforcement Learning with Quasi-Optimism
  7. 2025Exploration from a Primal-Dual Lens: Value-Incentivized Actor-Critic Methods for Sample-Efficient Online RL
  8. 2025Greedy Sampling Is Provably Efficient for RLHF