mlsanityData to 14 Oct 2026

Efficient Exploration at Scale

arXiv 2603.17378 ↗First posted 18 Mar 2026cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

We develop an online learning algorithm that dramatically improves the data efficiency of reinforcement learning from human feedback (RLHF). Our algorithm incrementally updates reward and language models as choice data is received. The reward model is fit to the choice data, while the language model is updated by a variation of reinforce, with reinforcement signals provided by the reward model. Several features enable the efficiency gains: a small affirmative nudge added to each reinforcement signal, an epistemic neural network that models reward uncertainty, and information-directed exploration. With Gemma large language models (LLMs), our algorithm matches the performance of offline RLHF trained on 200K labels using fewer than 20K labels, representing more than a 10x gain in data efficiency. Extrapolating from our results, we expect our algorithm trained on 1M labels to match offline RLHF trained on 1B labels. This represents a 1,000x gain. To our knowledge, these are the first results to demonstrate that such large improvements are possible.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 28 of its 33 references that are in the map. First posted on arXiv 18 Mar 2026.

Builds on

28 of 33 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Sample-Efficient Reinforcement Learning from Human Feedback via Information-Directed Sampling8 Feb
  2. 2024Learning from negative feedback, or positive feedback or both5 Oct
  3. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms5 Jun
  4. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF31 May
  5. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  6. 2024Active Preference Optimization for Sample Efficient RLHF16 Feb
  7. 2024Reinforcement Learning from Human Feedback with Active Queries14 Feb
  8. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 19

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  4. 2017Decoupled Weight Decay Regularizationcited by 1,831
  5. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  6. 2018AI safety via debatecited by 400
  7. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  8. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 219
  9. 2023Active Prompting with Chain-of-Thought for Large Language Modelscited by 154
  10. 2021Epistemic Neural Networkscited by 120
  11. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  12. 2024Active Preference Learning for Large Language Modelscited by 58
7 more
  1. 2024Efficient Exploration for LLMscited by 45
  2. 2023Sample Efficient Preference Alignment in LLMs via Active Explorationcited by 18
  3. 2024ActiveLLM: Large Language Model-based Active Learning for Textual Few-Shot Scenarioscited by 18
  4. 2024Sample-Efficient Alignment for LLMscited by 16
  5. 2025ActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignmentcited by 10
  6. 2024Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Explorationcited by 6
  7. 2025Sharpe Ratio-Guided Active Learning for Preference Optimization in RLHFcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback
  2. 2025Provably Efficient Online RLHF with One-Pass Reward Modeling
  3. 2026Data-dependent Exploration for Online Reinforcement Learning from Human Feedback
  4. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  5. 2025Toward Efficient Exploration by Large Language Model Agents
  6. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  7. 2025Avoiding mathbfexp(Rmax) scaling in RLHF through Preference-based Exploration
  8. 2025Is a Good Foundation Necessary for Efficient Reinforcement Learning? The Computational Role of the Base Model in Exploration