# Learning to Assist Humans without Inferring Rewards

arXiv: https://arxiv.org/abs/2411.02623 · HTML: https://mlsanity.com/paper/2411.02623/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 39 of its 66 references that are in the map.
- First posted on arXiv 4 Nov 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 30 of 39)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models](https://mlsanity.com/paper/2201.03544/) (2022)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress](https://mlsanity.com/paper/1806.06877/) (2018)
- [The Alignment Problem from a Deep Learning Perspective](https://mlsanity.com/paper/2209.00626/) (2022)
- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [Inverse Reward Design](https://mlsanity.com/paper/1711.02827/) (2017)
- [IQ-Learn: Inverse soft-Q Learning for Imitation](https://mlsanity.com/paper/2106.12142/) (2021)
- [Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning](https://mlsanity.com/paper/1509.08731/) (2015)
- [Optimal Policies Tend to Seek Power](https://mlsanity.com/paper/1912.01683/) (2019)
- [A survey on intrinsic motivation in reinforcement learning](https://mlsanity.com/paper/1908.06976/) (2019)
- [Consequences of Misaligned AI](https://mlsanity.com/paper/2102.03896/) (2021)
- [An elementary introduction to information geometry](https://mlsanity.com/paper/1808.08271/) (2018)
- [Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferences](https://mlsanity.com/paper/2006.14091/) (2020)
- [Empowerment -- an Introduction](https://mlsanity.com/paper/1310.1863/) (2013)
- [CIC: Contrastive Intrinsic Control for Unsupervised Skill Discovery](https://mlsanity.com/paper/2202.00161/) (2022)
- [Shared Autonomy via Deep Reinforcement Learning](https://mlsanity.com/paper/1802.01744/) (2018)
- [Lipschitz-constrained Unsupervised Skill Discovery](https://mlsanity.com/paper/2202.00914/) (2022)
- [Causal Confusion and Reward Misidentification in Preference-Based Reward Learning](https://mlsanity.com/paper/2204.06601/) (2022)

## Built on by (most cited first, 13 of 13)

- [Reward Models in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2506.15421/) (2025)
- [Cross-environment Cooperation Enables Zero-shot Multi-agent Coordination](https://mlsanity.com/paper/2504.12714/) (2025)
- [Reward Model Interpretability via Optimal and Pessimal Tokens](https://mlsanity.com/paper/2506.07326/) (2025)
- [Open-Ended Goal Inference through Actions and Language for Human-Robot Collaboration](https://mlsanity.com/paper/2512.04453/) (2025)
- [Active Inference and Human--Computer Interaction](https://mlsanity.com/paper/2412.14741/) (2024)
- [Estimating the Empowerment of Language Model Agents](https://mlsanity.com/paper/2509.22504/) (2025)
- [Training LLM Agents to Empower Humans](https://mlsanity.com/paper/2510.13709/) (2025)
- [Information-Theoretic Policy Pre-Training with Empowerment](https://mlsanity.com/paper/2510.05996/) (2025)
- [Temporal Representations for Exploration: Learning Complex Exploratory Behavior without Extrinsic Rewards](https://mlsanity.com/paper/2603.02008/) (2026)
- [Unifying Goal-Conditioned RL and Unsupervised Skill Learning via Control-Maximization](https://mlsanity.com/paper/2605.06145/) (2026)
- [When Assisting One Disempowers Another](https://mlsanity.com/paper/2511.04177/) (2025)
- [The Geometry of Empowerment](https://mlsanity.com/paper/2610.07796/) (2026)
- [A Fair Objective for Human-Empowerment-Preserving AI: Desiderata, Design, and Likely Behavioral Consequences](https://mlsanity.com/paper/2608.08240/) (2026)

## Abstract (arXiv)

Assistive agents should make humans' lives easier. Classically, such assistance is studied through the lens of inverse reinforcement learning, where an assistive agent (e.g., a chatbot, a robot) infers a human's intention and then selects actions to help the human reach that goal. This approach requires inferring intentions, which can be difficult in high-dimensional settings. We build upon prior work that studies assistance through the lens of empowerment: an assistive agent aims to maximize the influence of the human's actions such that they exert a greater control over the environmental outcomes and can solve tasks in fewer steps. We lift the major limitation of prior work in this area--scalability to high-dimensional settings--with contrastive successor representations. We formally prove that these representations estimate a similar notion of empowerment to that studied by prior work and provide a ready-made mechanism for optimizing it. Empirically, our proposed method outperforms prior methods on synthetic benchmarks, and scales to Overcooked, a cooperative game setting. Theoretically, our work connects ideas from information theory, neuroscience, and reinforcement learning, and charts a path for representations to play a critical role in solving assistive problems.