Learning Behaviors with Uncertain Human Feedback
On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF
Abstract (arXiv)
Human feedback is widely used to train agents in many domains. However, previous works rarely consider the uncertainty when humans provide feedback, especially in cases that the optimal actions are not obvious to the trainers. For example, the reward of a sub-optimal action can be stochastic and sometimes exceeds that of the optimal action, which is common in games or real-world. Trainers are likely to provide positive feedback to sub-optimal actions, negative feedback to the optimal actions and even do not provide feedback in some confusing situations. Existing works, which utilize the Expectation Maximization (EM) algorithm and treat the feedback model as hidden parameters, do not consider uncertainties in the learning environment and human feedback. To address this challenge, we introduce a novel feedback model that considers the uncertainty of human feedback. However, this incurs intractable calculus in the EM algorithm. To this end, we propose a novel approximate EM algorithm, in which we approximate the expectation step with the Gradient Descent method. Experimental results in both synthetic scenarios and two real-world scenarios with human participants demonstrate the superior performance of our proposed approach.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 4 of its 19 references that are in the map. First posted on arXiv 7 Jun 2020.
Builds on
4 of 19 references.
References · 4
- 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 238
- 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
- 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
- 2018Interactive Learning with Corrective Feedback for Policies based on Deep Neural Networkscited by 9
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
- 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons
- 2024CANDERE-COACH: Reinforcement Learning from Noisy Feedback
- 2024Adaptive Querying for Reward Learning from Human Feedback
- 2025Reinforcement Learning from Multi-level and Episodic Human Feedback
- 2024Learning from negative feedback, or positive feedback or both
- 2024Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Models
- 2025Sample-Efficient Reinforcement Learning from Human Feedback via Information-Directed Sampling