Batch Reinforcement Learning from Crowds
On the map: Offline RL and imitation learning from data → Offline RL algorithms
Abstract (arXiv)
A shortcoming of batch reinforcement learning is its requirement for rewards in data, thus not applicable to tasks without reward functions. Existing settings for lack of reward, such as behavioral cloning, rely on optimal demonstrations collected from humans. Unfortunately, extensive expertise is required for ensuring optimality, which hinder the acquisition of large-scale data for complex tasks. This paper addresses the lack of reward in a batch reinforcement learning setting by learning a reward function from preferences. Generating preferences only requires a basic understanding of a task. Being a mental process, generating preferences is faster than performing demonstrations. So preferences can be collected at scale from non-expert humans using crowdsourcing. This paper tackles a critical challenge that emerged when collecting data from non-expert humans: the noise in preferences. A novel probabilistic model is proposed for modelling the reliability of labels, which utilizes labels collaboratively. Moreover, the proposed model smooths the estimation with a learned reward function. Evaluation on Atari datasets demonstrates the effectiveness of the proposed model, followed by an ablation study to analyze the relative importance of the proposed ideas.
Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 11 of its 23 references that are in the map. First posted on arXiv 8 Nov 2021.
Builds on
11 of 23 references.
Closest ancestors · 1
Same direction, newest first
Other references · 10
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
- 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
- 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
- 2018Reward learning from human preferences and demonstrations in Ataricited by 314
- 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitationcited by 202
- 2019Imitation Learning from Imperfect Demonstrationcited by 112
- 2017Deep learning from crowdscited by 105
- 2019Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shiftcited by 47
- 2020Reducing Sampling Error in Batch Temporal Difference Learningcited by 11
Built on by
Citing papers per year: 2024: 4.
4 papers.
- 2024Crowd-PrefRL: Preference-Based Reward Learning from Crowdscited by 7
- 2024Scalable Interactive Machine Learning for Future Command and Controlcited by 2
- 2024VickreyFeedback: Cost-efficient Data Construction for Reinforcement Learning from Human Feedbackcited by 2
- 2024Online Policy Learning from Offline Preferencescited by 0
- 2024VickreyFeedback: Cost-efficient Data Construction for Reinforcement Learning from Human Feedback27 Sep 2024
- 2024Online Policy Learning from Offline Preferences15 Mar 2024
- 2024Scalable Interactive Machine Learning for Future Command and Control9 Feb 2024
- 2024Crowd-PrefRL: Preference-Based Reward Learning from Crowds17 Jan 2024
Similar, not cited
Close in wording, no citation either way
- 2023Reinforcement Learning from Diverse Human Preferences
- 2025Capturing Individual Human Preferences with Reward Features
- 2025Search-Based Credit Assignment for Offline Preference-Based Reinforcement Learning
- 2024PrefCLM: Enhancing Preference-based Reinforcement Learning with Crowdsourced Large Language Models ↗
- 2024Towards Comprehensive Preference Data Collection for Reward Modeling
- 2025Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Design
- 2025PB2: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning
- 2023A Strong Baseline for Batch Imitation Learning