# How Much Progress Did I Make? An Unexplored Human Feedback Signal for Teaching Robots

arXiv: https://arxiv.org/abs/2407.06459 · HTML: https://mlsanity.com/paper/2407.06459/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 20 of its 44 references that are in the map.
- First posted on arXiv 8 Jul 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 20 of 20)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Reward learning from human preferences and demonstrations in Atari](https://mlsanity.com/paper/1811.06521/) (2018)
- [Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces](https://mlsanity.com/paper/1709.10163/) (2017)
- [Imitation Learning from Imperfect Demonstration](https://mlsanity.com/paper/1901.09387/) (2019)
- [Using Natural Language for Reward Shaping in Reinforcement Learning](https://mlsanity.com/paper/1903.02020/) (2019)
- [Learning from Suboptimal Demonstration via Self-Supervised Reward Regression](https://mlsanity.com/paper/2010.11723/) (2020)
- [Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences](https://mlsanity.com/paper/2002.09089/) (2020)
- [Learning Reward Functions by Integrating Human Demonstrations and Preferences](https://mlsanity.com/paper/1906.08928/) (2019)
- [Deep Reinforcement Learning from Policy-Dependent Human Feedback](https://mlsanity.com/paper/1902.04257/) (2019)
- [DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback](https://mlsanity.com/paper/1810.11748/) (2018)
- [A Survey on Interactive Reinforcement Learning: Design Principles and Open Challenges](https://mlsanity.com/paper/2105.12949/) (2021)
- [The EMPATHIC Framework for Task Learning from Implicit Human Feedback](https://mlsanity.com/paper/2009.13649/) (2020)
- [Understanding Teacher Gaze Patterns for Robot Learning](https://mlsanity.com/paper/1907.07202/) (2019)
- [Uncertain Decisions Facilitate Better Preference Learning](https://mlsanity.com/paper/2106.10394/) (2021)
- [Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrations](https://mlsanity.com/paper/1907.03976/) (2019)
- [From "Thumbs Up" to "10 out of 10": Reconsidering Scalar Feedback in Interactive Reinforcement Learning](https://mlsanity.com/paper/2311.10284/) (2023)
- [Skill Acquisition via Automated Multi-Coordinate Cost Balancing](https://mlsanity.com/paper/1903.11725/) (2019)

## Built on by (most cited first, 2 of 2)

- [Demonstration Sidetracks: Categorizing Systematic Non-Optimality in Human Demonstrations](https://mlsanity.com/paper/2506.11262/) (2025)
- [PHIRL: Aligning Learned Rewards with Task Progress for Inverse Reinforcement Learning](https://mlsanity.com/paper/2609.31855/) (2026)

## Abstract (arXiv)

Enhancing the expressiveness of human teaching is vital for both improving robots' learning from humans and the human-teaching-robot experience. In this work, we characterize and test a little-used teaching signal: textitprogress, designed to represent the completion percentage of a task. We conducted two online studies with 76 crowd-sourced participants and one public space study with 40 non-expert participants to validate the capability of this progress signal. We find that progress indicates whether the task is successfully performed, reflects the degree of task completion, identifies unproductive but harmless behaviors, and is likely to be more consistent across participants. Furthermore, our results show that giving progress does not require extra workload and time. An additional contribution of our work is a dataset of 40 non-expert demonstrations from the public space study through an ice cream topping-adding task, which we observe to be multi-policy and sub-optimal, with sub-optimality not only from teleoperation errors but also from exploratory actions and attempts. The dataset is available at https://github.com/TeachingwithProgress/Non-ExpertDemonstrations.