# Decision Making for Human-in-the-loop Robotic Agents via Uncertainty-Aware Reinforcement Learning

arXiv: https://arxiv.org/abs/2303.06710 · HTML: https://mlsanity.com/paper/2303.06710/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 12 Mar 2023.

## Builds on (most cited first, 16 of 16)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Reinforcement Learning with Unsupervised Auxiliary Tasks](https://mlsanity.com/paper/1611.05397/) (2016)
- [Implicit Quantile Networks for Distributional Reinforcement Learning](https://mlsanity.com/paper/1806.06923/) (2018)
- [Randomized Prior Functions for Deep Reinforcement Learning](https://mlsanity.com/paper/1806.03335/) (2018)
- [Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning](https://mlsanity.com/paper/1509.08731/) (2015)
- [A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms](https://mlsanity.com/paper/1907.03146/) (2019)
- [Uncertainty-Aware Reinforcement Learning for Collision Avoidance](https://mlsanity.com/paper/1702.01182/) (2017)
- [DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback](https://mlsanity.com/paper/1810.11748/) (2018)
- [Reinforcement learning with human advice: a survey](https://mlsanity.com/paper/2005.11016/) (2020)
- [When to Ask for Help: Proactive Interventions in Autonomous Reinforcement Learning](https://mlsanity.com/paper/2210.10765/) (2022)
- [Ensemble Quantile Networks: Uncertainty-Aware Reinforcement Learning with Applications in Autonomous Driving](https://mlsanity.com/paper/2105.10266/) (2021)
- [Human-in-the-Loop Deep Reinforcement Learning with Application to Autonomous Driving](https://mlsanity.com/paper/2104.07246/) (2021)
- [Active Reward Learning from Multiple Teachers](https://mlsanity.com/paper/2303.00894/) (2023)

## Built on by (most cited first, 9 of 9)

- [Uncertainty Comes for Free: Human-in-the-Loop Policies with Diffusion Models](https://mlsanity.com/paper/2503.01876/) (2025)
- [Skill-Critic: Refining Learned Skills for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2306.08388/) (2023)
- [Simulating the Unseen: Crash Prediction Must Learn from What Did Not Happen](https://mlsanity.com/paper/2505.21743/) (2025)
- [MINT: Minimal Information Neuro-Symbolic Tree for Objective-Driven Knowledge-Gap Reasoning and Active Elicitation](https://mlsanity.com/paper/2602.05048/) (2026)
- [Enhancing Robot Navigation Policies with Task-Specific Uncertainty Managements](https://mlsanity.com/paper/2505.13837/) (2025)
- [Grower-in-the-Loop Interactive Reinforcement Learning for Greenhouse Climate Control](https://mlsanity.com/paper/2505.23355/) (2025)
- [HARP: Human-Assisted Regrouping with Permutation Invariant Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2409.11741/) (2024)
- [Language-guided Robust Navigation for Mobile Robots in Dynamically-changing Environments](https://mlsanity.com/paper/2409.19459/) (2024)
- [Policy-Calibrated DAgger: Offline Calibrated Noise Injection for Imitation Learning](https://mlsanity.com/paper/2609.30462/) (2026)

## Abstract (arXiv)

In a Human-in-the-Loop paradigm, a robotic agent is able to act mostly autonomously in solving a task, but can request help from an external expert when needed. However, knowing when to request such assistance is critical: too few requests can lead to the robot making mistakes, but too many requests can overload the expert. In this paper, we present a Reinforcement Learning based approach to this problem, where a semi-autonomous agent asks for external assistance when it has low confidence in the eventual success of the task. The confidence level is computed by estimating the variance of the return from the current state. We show that this estimate can be iteratively improved during training using a Bellman-like recursion. On discrete navigation problems with both fully- and partially-observable state information, we show that our method makes effective use of a limited budget of expert calls at run-time, despite having no access to the expert at training time.