mlsanityData to 7 Oct 2026

Information Directed Reward Learning for Reinforcement Learning

arXiv 2102.12466 ↗First posted 24 Feb 2021cs.LGcited by 24

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

For many reinforcement learning (RL) applications, specifying a reward is difficult. This paper considers an RL setting where the agent obtains information about the reward only by querying an expert that can, for example, evaluate individual states or provide binary preferences over trajectories. From such expensive feedback, we aim to learn a model of the reward that allows standard RL algorithms to achieve high expected returns with as few expert queries as possible. To this end, we propose Information Directed Reward Learning (IDRL), which uses a Bayesian model of the reward and selects queries that maximize the information gain about the difference in return between plausibly optimal policies. In contrast to prior active reward learning methods designed for specific types of queries, IDRL naturally accommodates different query types. Moreover, it achieves similar or better performance with significantly fewer queries by shifting the focus from reducing the reward approximation error to improving the policy induced by the reward model. We support our findings with extensive evaluations in multiple environments and with different query types.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 19 of its 59 references that are in the map. First posted on arXiv 24 Feb 2021.

Builds on

19 of 59 references.

Closest ancestors · 4

Same direction, newest first

  1. 2020Active Preference-Based Gaussian Process Regression for Reward Learning6 May
  2. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  3. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  4. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 15

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  4. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  5. 2015Scalable Bayesian Optimization Using Deep Neural Networkscited by 284
  6. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
  7. 2018When Gaussian Process Meets Big Data: A Review of Scalable GPscited by 205
  8. 2014Learning to Optimize via Information-Directed Samplingcited by 168
  9. 2018The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigationcited by 159
  10. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  11. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
  12. 2019Sequential Experimental Design for Transductive Linear Banditscited by 88
3 more
  1. 2019Asking Easy Questions: A User-Friendly Approach to Active Reward Learningcited by 67
  2. 2020Information Directed Sampling for Linear Partial Monitoringcited by 39
  3. 2020Active Preference Learning using Maximum Regretcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 4; 2023: 4; 2024: 5; 2025: 9; 2026: 1.

24 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  2. 2023Submodular Reinforcement Learningcited by 23
  3. 2022Active Exploration via Experiment Design in Markov Chainscited by 19
  4. 2021An Experimental Design Perspective on Model-Based Reinforcement Learningcited by 16
  5. 2023Provably Feedback-Efficient Reinforcement Learning via Active Reward Learningcited by 15
  6. 2025Provable Maximum Entropy Manifold Exploration via Diffusion Modelscited by 12
  7. 2023CLUE: Calibrated Latent Guidance for Offline Reinforcement Learningcited by 11
  8. 2025SPARTA ALIGNMENT: Collectively Aligning Multiple Language Models through Combatcited by 8
  9. 2024Directed Exploration in Reinforcement Learning from Linear Temporal Logiccited by 7
  10. 2024Beyond Optimism: Exploration With Partially Observable Rewardscited by 6
  11. 2025Optimistic Task Inference for Behavior Foundation Modelscited by 5
  12. 2022Environment Design for Inverse Reinforcement Learningcited by 4
12 more
  1. 2024Preference Elicitation for Offline Reinforcement Learningcited by 3
  2. 2022Reward Learning with Trees: Methods and Evaluationcited by 2
  3. 2022Actively Learning Costly Reward Functions for Reinforcement Learningcited by 2
  4. 2024A Generalized Acquisition Function for Preference-based Reward Learningcited by 1
  5. 2024PyGen: A Collaborative Human-AI Approach to Python Package Creationcited by 1
  6. 2025Active Measuring in Reinforcement Learning With Delayed Negative Effectscited by 1
  7. 2023Learning Interpretable Models of Aircraft Handling Behaviour by Reinforcement Learning from Human Feedbackcited by 0
  8. 2025Empirical Bound Information-Directed Sampling for Norm-Agnostic Banditscited by 0
  9. 2025ASkDAgger: Active Skill-level Data Aggregation for Interactive Imitation Learningcited by 0
  10. 2025Which Rewards Matter? Reward Selection for Reinforcement Learning under Limited Feedbackcited by 0
  11. 2025Efficient Personalization of Generative Models via Optimal Experimental Designcited by 0
  12. 2026OPRIDE: Offline Preference-based Reinforcement Learning via In-Dataset Explorationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Bayesian Reparameterization of Reward-Conditioned Reinforcement Learning with Energy-based Models
  2. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learning
  3. 2025Sample-Efficient Reinforcement Learning from Human Feedback via Information-Directed Sampling
  4. 2025Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback
  5. 2025Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Design
  6. 2024The Distributional Reward Critic Framework for Reinforcement Learning Under Perturbed Rewards
  7. 2026Information-Directed Offline-to-Online Reinforcement Learning
  8. 2024Bayesian Inverse Reinforcement Learning for Non-Markovian Rewards