# Learning under Misspecified Objective Spaces

arXiv: https://arxiv.org/abs/1810.05157 · HTML: https://mlsanity.com/paper/1810.05157/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Oct 2018.

## Built on by (most cited first, 14 of 14)

- [Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences](https://mlsanity.com/paper/2002.09089/) (2020)
- [Inducing Structure in Reward Learning by Learning Features](https://mlsanity.com/paper/2201.07082/) (2022)
- [Diagnosis, Feedback, Adaptation: A Human-in-the-Loop Framework for Test-Time Policy Adaptation](https://mlsanity.com/paper/2307.06333/) (2023)
- [Autonomous Robotic Reinforcement Learning with Asynchronous Human Feedback](https://mlsanity.com/paper/2310.20608/) (2023)
- [Aligning Robot and Human Representations](https://mlsanity.com/paper/2302.01928/) (2023)
- [Analyzing Human Models that Adapt Online](https://mlsanity.com/paper/2103.05746/) (2021)
- [Auditing Robot Learning for Safety and Compliance during Deployment](https://mlsanity.com/paper/2110.05702/) (2021)
- [When Life Gives You BC, Make Q-functions: Extracting Q-values from Behavior Cloning for On-Robot Reinforcement Learning](https://mlsanity.com/paper/2605.05172/) (2026)
- [Learning from Physical Human Feedback: An Object-Centric One-Shot Adaptation Method](https://mlsanity.com/paper/2203.04951/) (2022)
- [MEReQ: Max-Ent Residual-Q Inverse RL for Sample-Efficient Alignment from Intervention](https://mlsanity.com/paper/2406.16258/) (2024)
- [Enhancing Goal Inference via Correction Timing](https://mlsanity.com/paper/2602.18603/) (2026)
- [Aligning Robot Representations with Humans](https://mlsanity.com/paper/2205.07882/) (2022)
- [QuickLAP: Quick Language-Action Preference Learning for Semi-Autonomous Agents](https://mlsanity.com/paper/2511.17855/) (2025)
- [Robots That Know What to Ask: Recovering Misaligned Rewards through Targeted Explanations](https://mlsanity.com/paper/2605.22986/) (2026)

## Abstract (arXiv)

Learning robot objective functions from human input has become increasingly important, but state-of-the-art techniques assume that the human's desired objective lies within the robot's hypothesis space. When this is not true, even methods that keep track of uncertainty over the objective fail because they reason about which hypothesis might be correct, and not whether any of the hypotheses are correct. We focus specifically on learning from physical human corrections during the robot's task execution, where not having a rich enough hypothesis space leads to the robot updating its objective in ways that the person did not actually intend. We observe that such corrections appear irrelevant to the robot, because they are not the best way of achieving any of the candidate objectives. Instead of naively trusting and learning from every human interaction, we propose robots learn conservatively by reasoning in real time about how relevant the human's correction is for the robot's hypothesis space. We test our inference method in an experiment with human interaction data, and demonstrate that this alleviates unintended learning in an in-person user study with a 7DoF robot manipulator.