mlsanityData to 14 Oct 2026

Learning Reward Functions from Scale Feedback

arXiv 2110.00284 ↗First posted 1 Oct 2021cs.RO cs.AI cs.LGcited by 24

Not on the map. Why?

Abstract (arXiv)

Today's robots are increasingly interacting with people and need to efficiently learn inexperienced user's preferences. A common framework is to iteratively query the user about which of two presented robot trajectories they prefer. While this minimizes the users effort, a strict choice does not yield any information on how much one trajectory is preferred. We propose scale feedback, where the user utilizes a slider to give more nuanced information. We introduce a probabilistic model on how users would provide feedback and derive a learning framework for the robot. We demonstrate the performance benefit of slider feedback in simulations, and validate our approach in two user studies suggesting that scale feedback enables more effective learning in practice.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 16 of its 37 references that are in the map. First posted on arXiv 1 Oct 2021.

Builds on

16 of 37 references.

References · 16

  1. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
  2. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 120
  3. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regressioncited by 84
  4. 2020Active Preference-Based Gaussian Process Regression for Reward Learningcited by 71
  5. 2019Asking Easy Questions: A User-Friendly Approach to Active Reward Learningcited by 68
  6. 2019Learning Reward Functions by Integrating Human Demonstrations and Preferencescited by 64
  7. 2021Learning Multimodal Rewards from Rankingscited by 41
  8. 2019Controlling Assistive Robots with Learned Latent Actionscited by 28
  9. 2020Active Preference Learning using Maximum Regretcited by 21
  10. 2021Learning Human Objectives from Sequences of Physical Correctionscited by 21
  11. 2020ROIAL: Region of Interest Active Learning for Characterizing Exoskeleton Gait Preference Landscapescited by 20
  12. 2018Learning from Richer Human Guidance: Augmenting Comparison-Based Learning with Feature Queriescited by 16
4 more
  1. 2018Including Uncertainty when Learning from Human Correctionscited by 15
  2. 2020Interactive Robot Training for Non-Markov Taskscited by 9
  3. 2019Improving User Specifications for Robot Behavior through Active Preference Learning: Framework and Evaluationcited by 7
  4. 2019Bayesian Active Learning for Collaborative Task Specification Using Equivalence Regionscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 13; 2025: 5; 2026: 3.

24 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  2. 2024TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correctioncited by 59
  3. 2024Mapping Social Choice Theory to RLHFcited by 27
  4. 2023Active Reward Learning from Online Preferencescited by 13
  5. 2024ViSaRL: Visual Reinforcement Learning Guided by Human Saliencycited by 12
  6. 2024Batch Active Learning of Reward Functions from Human Preferencescited by 8
  7. 2024Crowd-PrefRL: Preference-Based Reward Learning from Crowdscited by 7
  8. 2024Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learningcited by 6
  9. 2023Synthetic Imitation Edit Feedback for Factual Alignment in Clinical Summarizationcited by 5
  10. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Frameworkcited by 5
  11. 2024Enhancing Preference-based Linear Bandits via Human Response Timecited by 3
  12. 2026RDA: Reward Design Agent for Reinforcement Learningcited by 3
12 more
  1. 2025Interactive Multi-Objective Probabilistic Preference Learning with Soft and Hard Boundscited by 2
  2. 2024A Generalized Acquisition Function for Preference-based Reward Learningcited by 1
  3. 2024FARPLS: A Feature-Augmented Robot Trajectory Preference Labeling System to Assist Human Labelers' Preference Elicitationcited by 1
  4. 2024MEReQ: Max-Ent Residual-Q Inverse RL for Sample-Efficient Alignment from Interventioncited by 1
  5. 2025Preference Learning with Response Time: Robust Losses and Guaranteescited by 1
  6. 2025Beyond Ordinal Preferences: Why Alignment Needs Cardinal Human Feedbackcited by 1
  7. 2025Learning from Preferences and Mixed Demonstrations in General Settingscited by 1
  8. 2024Learning to Communicate Functional States with Nonverbal Expressions for Improved Human-Robot Collaborationcited by 0
  9. 2024Representation Alignment from Human Feedback for Cross-Embodiment Reward Learning from Mixed-Quality Demonstrationscited by 0
  10. 2025Granular feedback merits sophisticated aggregationcited by 0
  11. 2026Reward Learning through Ranking Mean Squared Errorcited by 0
  12. 2026Action- and Language-Conditioned Video Assessment for Embodied Controlcited by 0