# Aligning Robot Representations with Humans

arXiv: https://arxiv.org/abs/2205.07882 · HTML: https://mlsanity.com/paper/2205.07882/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 32 of its 55 references that are in the map.
- First posted on arXiv 15 May 2022.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [CLIPort: What and Where Pathways for Robotic Manipulation](https://mlsanity.com/paper/2109.12098/) (2021)
- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [An Algorithmic Perspective on Imitation Learning](https://mlsanity.com/paper/1811.06711/) (2018)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [Causal Confusion in Imitation Learning](https://mlsanity.com/paper/1905.11979/) (2019)
- [Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations](https://mlsanity.com/paper/1904.06387/) (2019)
- [Inverse Reward Design](https://mlsanity.com/paper/1711.02827/) (2017)
- [Zero-Shot Visual Imitation](https://mlsanity.com/paper/1804.08606/) (2018)
- [Playing hard exploration games by watching YouTube](https://mlsanity.com/paper/1805.11592/) (2018)
- [Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences](https://mlsanity.com/paper/2002.09089/) (2020)
- [Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition](https://mlsanity.com/paper/1805.11686/) (2018)
- [Learning a Prior over Intent via Meta-Inverse Reinforcement Learning](https://mlsanity.com/paper/1805.12573/) (2018)
- [Watch This: Scalable Cost-Function Learning for Path Planning in Urban Environments](https://mlsanity.com/paper/1607.02329/) (2016)
- [Learning Preferences for Manipulation Tasks from Online Coactive Feedback](https://mlsanity.com/paper/1601.00741/) (2016)
- [Feature Expansive Reward Learning: Rethinking Human Input](https://mlsanity.com/paper/2006.13208/) (2020)
- [Quantifying Hypothesis Space Misspecification in Learning from Human-Robot Demonstrations and Physical Corrections](https://mlsanity.com/paper/2002.00941/) (2020)
- [LESS is More: Rethinking Probabilistic Models of Human Behavior](https://mlsanity.com/paper/2001.04465/) (2020)
- [Inducing Structure in Reward Learning by Learning Features](https://mlsanity.com/paper/2201.07082/) (2022)
- [Including Uncertainty when Learning from Human Corrections](https://mlsanity.com/paper/1806.02454/) (2018)
- [Situational Confidence Assistance for Lifelong Shared Autonomy](https://mlsanity.com/paper/2104.06556/) (2021)
- [Learning under Misspecified Objective Spaces](https://mlsanity.com/paper/1810.05157/) (2018)
- [On complementing end-to-end human behavior predictors with planning](https://mlsanity.com/paper/2103.05661/) (2021)

## Abstract (arXiv)

As robots are increasingly deployed in real-world scenarios, a key question is how to best transfer knowledge learned in one environment to another, where shifting constraints and human preferences render adaptation challenging. A central challenge remains that often, it is difficult (perhaps even impossible) to capture the full complexity of the deployment environment, and therefore the desired tasks, at training time. Consequently, the representation, or abstraction, of the tasks the human hopes for the robot to perform in one environment may be misaligned with the representation of the tasks that the robot has learned in another. We postulate that because humans will be the ultimate evaluator of system success in the world, they are best suited to communicating the aspects of the tasks that matter to the robot. Our key insight is that effective learning from human input requires first explicitly learning good intermediate representations and then using those representations for solving downstream tasks. We highlight three areas where we can use this approach to build interactive systems and offer future directions of work to better create advanced collaborative robots.