mlsanityData to 7 Oct 2026

How Much Progress Did I Make? An Unexplored Human Feedback Signal for Teaching Robots

arXiv 2407.06459 ↗First posted 8 Jul 2024cs.RO cs.AI cs.LGcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Enhancing the expressiveness of human teaching is vital for both improving robots' learning from humans and the human-teaching-robot experience. In this work, we characterize and test a little-used teaching signal: textitprogress, designed to represent the completion percentage of a task. We conducted two online studies with 76 crowd-sourced participants and one public space study with 40 non-expert participants to validate the capability of this progress signal. We find that progress indicates whether the task is successfully performed, reflects the degree of task completion, identifies unproductive but harmless behaviors, and is likely to be more consistent across participants. Furthermore, our results show that giving progress does not require extra workload and time. An additional contribution of our work is a dataset of 40 non-expert demonstrations from the public space study through an ice cream topping-adding task, which we observe to be multi-policy and sub-optimal, with sub-optimality not only from teleoperation errors but also from exploratory actions and attempts. The dataset is available at https://github.com/TeachingwithProgress/Non-ExpertDemonstrations.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 20 of its 44 references that are in the map. First posted on arXiv 8 Jul 2024.

Builds on

20 of 44 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023From "Thumbs Up" to "10 out of 10": Reconsidering Scalar Feedback in Interactive Reinforcement Learning17 Nov
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  3. 2020The EMPATHIC Framework for Task Learning from Implicit Human Feedback28 Sep
  4. 2019Deep Reinforcement Learning from Policy-Dependent Human Feedback12 Feb
  5. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  6. 2018DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback28 Oct

Other references · 14

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2016Generative Adversarial Imitation Learningcited by 1,576
  3. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  4. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  5. 2019Using Natural Language for Reward Shaping in Reinforcement Learningcited by 114
  6. 2019Imitation Learning from Imperfect Demonstrationcited by 112
  7. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regressioncited by 84
  8. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  9. 2019Learning Reward Functions by Integrating Human Demonstrations and Preferencescited by 64
  10. 2021A Survey on Interactive Reinforcement Learning: Design Principles and Open Challengescited by 41
  11. 2019Understanding Teacher Gaze Patterns for Robot Learningcited by 12
  12. 2021Uncertain Decisions Facilitate Better Preference Learningcited by 10
2 more
  1. 2019Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrationscited by 5
  2. 2019Skill Acquisition via Automated Multi-Coordinate Cost Balancingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Demonstration Sidetracks: Categorizing Systematic Non-Optimality in Human Demonstrationscited by 2
  2. 2026PHIRL: Aligning Learned Rewards with Task Progress for Inverse Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Progress Reward Modeling for Robotic Learning: A Comprehensive Survey
  2. 2026ProgressCompass: Embodied Progress Reward Models Are Lost Without the Right Context
  3. 2024Consistency Matters: Defining Demonstration Data Quality Metrics in Robot Learning from Demonstration
  4. 2019Benchmark for Skill Learning from Demonstration: Impact of User Experience, Task Complexity, and Start Configuration on Performance
  5. 2026TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics
  6. 2020Investigating the Effects of Robot Engagement Communication on Learning from Demonstration
  7. 2025Sequentially Teaching Sequential Tasks (ST)2: Teaching Robots Long-horizon Manipulation Skills
  8. 2026UR-VC: Unsupervised Robotic Value Correction for Time-Derived Progress Proxies