# CueLearner: Bootstrapping and local policy adaptation from relative feedback

arXiv: https://arxiv.org/abs/2507.04730 · HTML: https://mlsanity.com/paper/2507.04730/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods.
- Builds on 15 of its 28 references that are in the map.
- First posted on arXiv 7 Jul 2025.

## Placed in

- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 15 of 15)

- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training](https://mlsanity.com/paper/2106.05091/) (2021)
- [Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments](https://mlsanity.com/paper/2301.04195/) (2023)
- [Interactive Learning from Policy-Dependent Human Feedback](https://mlsanity.com/paper/1701.06049/) (2017)
- [Yell At Your Robot: Improving On-the-Fly from Language Corrections](https://mlsanity.com/paper/2403.12910/) (2024)
- [Few-Shot Preference Learning for Human-in-the-Loop RL](https://mlsanity.com/paper/2212.03363/) (2022)
- [MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/2212.05698/) (2022)
- ["No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomy](https://mlsanity.com/paper/2301.02555/) (2023)
- [DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback](https://mlsanity.com/paper/1810.11748/) (2018)
- [Interactive Robot Learning from Verbal Correction](https://mlsanity.com/paper/2310.17555/) (2023)
- [Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Dense and Sparse Reward Environments](https://mlsanity.com/paper/1910.04281/) (2019)
- [Interactive Learning with Corrective Feedback for Policies based on Deep Neural Networks](https://mlsanity.com/paper/1810.00466/) (2018)
- [Action Advising with Advice Imitation in Deep Reinforcement Learning](https://mlsanity.com/paper/2104.08441/) (2021)

## Abstract (arXiv)

Human guidance has emerged as a powerful tool for enhancing reinforcement learning (RL). However, conventional forms of guidance such as demonstrations or binary scalar feedback can be challenging to collect or have low information content, motivating the exploration of other forms of human input. Among these, relative feedback (i.e., feedback on how to improve an action, such as "more to the left") offers a good balance between usability and information richness. Previous research has shown that relative feedback can be used to enhance policy search methods. However, these efforts have been limited to specific policy classes and use feedback inefficiently. In this work, we introduce a novel method to learn from relative feedback and combine it with off-policy reinforcement learning. Through evaluations on two sparse-reward tasks, we demonstrate our method can be used to improve the sample efficiency of reinforcement learning by guiding its exploration process. Additionally, we show it can adapt a policy to changes in the environment or the user's preferences. Finally, we demonstrate real-world applicability by employing our approach to learn a navigation policy in a sparse reward setting.