mlsanityData to 14 Oct 2026

Opinion-Guided Reinforcement Learning

arXiv 2405.17287 ↗First posted 27 May 2024cs.LG cs.AIcited by 3

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Human guidance is often desired in reinforcement learning to improve the performance of the learning agent. However, human insights are often mere opinions and educated guesses rather than well-formulated arguments. While opinions are subject to uncertainty, e.g., due to partial informedness or ignorance about a problem, they also emerge earlier than hard evidence can be produced. Thus, guiding reinforcement learning agents by way of opinions offers the potential for more performant learning processes, but comes with the challenge of modeling and managing opinions in a formal way. In this article, we present a method to guide reinforcement learning agents through opinions. To this end, we provide an end-to-end method to model and manage advisors' opinions. To assess the utility of the approach, we evaluate it with synthetic (oracle) and human advisors, at different levels of uncertainty, and under multiple advice strategies. Our results indicate that opinions, even if uncertain, improve the performance of reinforcement learning agents, resulting in higher rewards, more efficient exploration, and a better reinforced policy. Although we demonstrate our approach through a two-dimensional topological running example, our approach is applicable to complex problems with higher dimensions as well.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 12 of its 66 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

12 of 66 references.

References · 12

  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 913
  3. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  4. 2017Safe Reinforcement Learning via Shieldingcited by 449
  5. 2021Demonstration-Guided Reinforcement Learning with Learned Skillscited by 76
  6. 2022Interactive Imitation Learning in Robotics: A Surveycited by 55
  7. 2018DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedbackcited by 41
  8. 2018Managing engineering systems with large state and action spaces through deep reinforcement learningcited by 32
  9. 2020Reinforcement learning with human advice: a surveycited by 24
  10. 2020Learning a Low-dimensional Representation of a Safe Region for Safe Reinforcement Learning on Dynamical Systemscited by 6
  11. 2020Using Subjective Logic to Estimate Uncertainty in Multi-Armed Bandit Problemscited by 2
  12. 2023Uncertainty-Aware Reward-based Deep Reinforcement Learning for Intent Analysis of Social Media Informationcited by 1

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025AI Simulation by Digital Twins: Systematic Survey, Reference Framework, and Mapping to a Standardized Architecturecited by 6
  2. 2025Complex Model Transformations by Reinforcement Learning with Uncertain Human Guidancecited by 2
  3. 2025Human-Allied Relational Reinforcement Learningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Models
  2. 2024Enhancing Reinforcement Learning Agents with Local Guides
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  4. 2024GUIDE: Real-Time Human-Shaped Agents
  5. 2025Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback
  6. 2023Query-Policy Misalignment in Preference-Based Reinforcement Learning
  7. 2025PB2: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning
  8. 2025The Limits of Preference Data for Post-Training