# FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback

arXiv: https://arxiv.org/abs/2001.06781 · HTML: https://mlsanity.com/paper/2001.06781/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jan 2020.

## Built on by (most cited first, 15 of 15)

- [CREW: Facilitating Human-AI Teaming Research](https://mlsanity.com/paper/2408.00170/) (2024)
- [Advice Conformance Verification by Reinforcement Learning agents for Human-in-the-Loop](https://mlsanity.com/paper/2210.03455/) (2022)
- [Accelerating Reinforcement Learning Agent with EEG-based Implicit Human Feedback](https://mlsanity.com/paper/2006.16498/) (2020)
- [Shaping Advice in Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2103.15941/) (2021)
- [Informativeness of Reward Functions in Reinforcement Learning](https://mlsanity.com/paper/2402.07019/) (2024)
- [From Sparse to Dense: Toddler-inspired Reward Transition in Goal-Oriented Reinforcement Learning](https://mlsanity.com/paper/2501.17842/) (2025)
- [Zero-Shot LLMs in Human-in-the-Loop RL: Replacing Human Feedback for Reward Shaping](https://mlsanity.com/paper/2503.22723/) (2025)
- [Towards Interactive Reinforcement Learning with Intrinsic Feedback](https://mlsanity.com/paper/2112.01575/) (2021)
- [Iterative Reward Shaping using Human Feedback for Correcting Reward Misspecification](https://mlsanity.com/paper/2308.15969/) (2023)
- [Privacy-Preserving Reinforcement Learning Beyond Expectation](https://mlsanity.com/paper/2203.10165/) (2022)
- [Reinforcement Learning for UAV control with Policy and Reward Shaping](https://mlsanity.com/paper/2212.03828/) (2022)
- [Contextual bandits with entropy-based human feedback](https://mlsanity.com/paper/2502.08759/) (2025)
- [Shaping Advice in Deep Reinforcement Learning](https://mlsanity.com/paper/2202.09489/) (2022)
- [On the Effect of Robot Errors on Human Teaching Dynamics](https://mlsanity.com/paper/2409.09827/) (2024)
- [Where to Intervene: Action Selection in Deep Reinforcement Learning](https://mlsanity.com/paper/2507.04187/) (2025)

## Abstract (arXiv)

Reinforcement learning has been successful in training autonomous agents to accomplish goals in complex environments. Although this has been adapted to multiple settings, including robotics and computer games, human players often find it easier to obtain higher rewards in some environments than reinforcement learning algorithms. This is especially true of high-dimensional state spaces where the reward obtained by the agent is sparse or extremely delayed. In this paper, we seek to effectively integrate feedback signals supplied by a human operator with deep reinforcement learning algorithms in high-dimensional state spaces. We call this FRESH (Feedback-based REward SHaping). During training, a human operator is presented with trajectories from a replay buffer and then provides feedback on states and actions in the trajectory. In order to generalize feedback signals provided by the human operator to previously unseen states and actions at test-time, we use a feedback neural network. We use an ensemble of neural networks with a shared network architecture to represent model uncertainty and the confidence of the neural network in its output. The output of the feedback neural network is converted to a shaping reward that is augmented to the reward provided by the environment. We evaluate our approach on the Bowling and Skiing Atari games in the arcade learning environment. Although human experts have been able to achieve high scores in these environments, state-of-the-art deep learning algorithms perform poorly. We observe that FRESH is able to achieve much higher scores than state-of-the-art deep learning algorithms in both environments. FRESH also achieves a 21.4% higher score than a human expert in Bowling and does as well as a human expert in Skiing.