Data to 14 Oct 2026

Learning Multimodal Rewards from Rankings

arXiv 2109.12750 ↗First posted 27 Sep 2021cs.LG cs.AI cs.ROcited by 41

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Learning from human feedback has shown to be a useful approach in acquiring robot reward functions. However, expert feedback is often assumed to be drawn from an underlying unimodal reward function. This assumption does not always hold including in settings where multiple experts provide data or when a single expert provides data for different tasks -- we thus go beyond learning a unimodal reward and focus on learning a multimodal reward function. We formulate the multimodal reward learning as a mixture learning problem and develop a novel ranking-based learning approach, where the experts are only required to rank a given set of trajectories. Furthermore, as access to interaction data is often expensive in robotics, we develop an active querying approach to accelerate the learning process. We conduct experiments and user studies using a multi-task variant of OpenAI's LunarLander and a real Fetch robot, where we collect data from multiple users with different preferences. The results suggest that our approach can efficiently learn multimodal reward functions, and improve data-efficiency over benchmark methods that we adapt to our learning problem.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 67 references that are in the map. First posted on arXiv 27 Sep 2021.

Builds on

26 of 67 references.

Closest ancestors · 2

Same direction, newest first

Other references · 24

2013Playing Atari with Deep Reinforcement Learningcited by 3,274
2016OpenAI Gymcited by 1,736
2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
2019Imitation Learning from Imperfect Demonstrationcited by 114
2018Batch Active Preference-Based Learning of Reward Functionscited by 87
2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
2019Asking Easy Questions: A User-Friendly Approach to Active Reward Learningcited by 68
2017Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Netscited by 64
2020When Humans Aren't Optimal: Robots that Collaborate with Risk-Aware Humanscited by 34
2019Preference-Based Learning for Exoskeleton Gait Optimizationcited by 24
2020Reinforcement Learning based Control of Imitative Policies for Near-Accident Drivingcited by 23
2020Active Preference Learning using Maximum Regretcited by 21

12 more

2021Learning Human Objectives from Sequences of Physical Correctionscited by 21
2019Composing Task-Agnostic Policies with Deep Reinforcement Learningcited by 20
2020ROIAL: Region of Interest Active Learning for Characterizing Exoskeleton Gait Preference Landscapescited by 20
2016Learning Mixtures of Plackett-Luce Modelscited by 19
2016A First Look at Ad-block Detection: A New Arms Race on the Webcited by 16
2017Simultaneous Policy Learning and Latent State Inference for Imitating Driver Behaviorcited by 16
2018Learning from Richer Human Guidance: Augmenting Comparison-Based Learning with Feature Queriescited by 16
2019The Green Choice: Learning and Influencing Human Decisions on Shared Roadscited by 14
2018Efficiently Learning Mixtures of Mallows Modelscited by 13
2017A Nearly Instance Optimal Algorithm for Top-k Ranking under the Multinomial Logit Modelcited by 12
2020Triple-GAIL: A Multi-Modal Imitation Learning Framework with Generative Adversarial Netscited by 11
2020Interactive Robot Training for Non-Markov Taskscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 5; 2023: 9; 2024: 10; 2025: 8; 2026: 7.

41 papers.

Sort:

By area:

2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 156
2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 134
2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 87
2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 60
2022Probabilistic Conformal Prediction Using Conditional Random Samplescited by 47
2023Promptable Behaviors: Personalizing Multi-Objective Rewards from Human Preferencescited by 26
2021MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learningcited by 25
2021Learning Reward Functions from Scale Feedbackcited by 24
2022Eliciting Compatible Demonstrations for Multi-Human Imitation Learningcited by 23
2022Dexterous Manipulation from Images: Autonomous Real-World RL via Substep Guidancecited by 23

29 more

2023REBOOT: Reuse Data for Bootstrapping Efficient Real-World Dexterous Manipulationcited by 15
2024Offline Imitation Learning Through Graph Search and Retrievalcited by 14
2024Listwise Reward Estimation for Offline Preference-based Reinforcement Learningcited by 14
2023Active Reward Learning from Online Preferencescited by 13
2024Learning to Assist Humans without Inferring Rewardscited by 13
2024Teaching Language Models to Self-Improve by Learning from Language Feedbackcited by 9
2024Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learningcited by 6
2022Learning from Imperfect Demonstrations via Adversarial Confidence Transfercited by 5
2024Learning Reward and Policy Jointly from Demonstration and Preference Improves Alignmentcited by 4
2023Curriculum-Based Imitation of Versatile Skillscited by 3
2025Contrastive Learning from Exploratory Actions: Leveraging Natural Interactions for Preference Elicitationcited by 3
2026RDA: Reward Design Agent for Reinforcement Learningcited by 3
2025Robust Reward Alignment via Hypothesis Space Batch Cuttingcited by 2
2025AED: Automatic Discovery of Effective and Diverse Vulnerabilities for Autonomous Driving Policy with Large Language Modelscited by 2
2025Interactive Multi-Objective Probabilistic Preference Learning with Soft and Hard Boundscited by 2
2023Boosting Feedback Efficiency of Interactive Reinforcement Learning by Adaptive Learning from Scorescited by 1
2023Active Inverse Learning in Stackelberg Trajectory Gamescited by 1
2024A Generalized Acquisition Function for Preference-based Reward Learningcited by 1
2024Improving User Experience in Preference-Based Optimization of Reward Functions for Assistive Robotscited by 1
2025Efficiently Generating Expressive Quadruped Behaviors via Language-Guided Preference Learningcited by 1
2025Learning from Preferences and Mixed Demonstrations in General Settingscited by 1
2025Multimodal Bandits: Regret Lower Bounds and Optimal Algorithmscited by 1
2026Multi-Modal, Multi-Environment Machine Teaching for Robust Reward Learningcited by 1
2024Representation Alignment from Human Feedback for Cross-Embodiment Reward Learning from Mixed-Quality Demonstrationscited by 0
2025Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrationscited by 0
2026MAVRL: Learning Reward Functions from Multiple Feedback Types with Amortized Variational Inferencecited by 0
2026Quantifying Potential Observation Missingness in Inverse Reinforcement Learningcited by 0
2026PAWS: Preference Learning with Advantage-Weighted Segmentscited by 0
2026Subspace Inference Enables Efficient Active Reward Learning from Preferencescited by 0

Similar, not cited

Close in wording, no citation either way