# Zero-Shot Assistance in Sequential Decision Problems

arXiv: https://arxiv.org/abs/2202.07364 · HTML: https://mlsanity.com/paper/2202.07364/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 40 references that are in the map.
- First posted on arXiv 15 Feb 2022.

## Builds on (most cited first, 14 of 14)

- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress](https://mlsanity.com/paper/1806.06877/) (2018)
- [Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations](https://mlsanity.com/paper/1904.06387/) (2019)
- ["Other-Play" for Zero-Shot Coordination](https://mlsanity.com/paper/2003.02979/) (2020)
- [Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces](https://mlsanity.com/paper/1709.10163/) (2017)
- [Shared Autonomy via Deep Reinforcement Learning](https://mlsanity.com/paper/1802.01744/) (2018)
- [On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inference](https://mlsanity.com/paper/1906.09624/) (2019)
- [Shared Autonomy via Hindsight Optimization](https://mlsanity.com/paper/1503.07619/) (2015)
- [Learning the Preferences of Ignorant, Inconsistent Agents](https://mlsanity.com/paper/1512.05832/) (2015)
- [Occam's razor is insufficient to infer the preferences of irrational agents](https://mlsanity.com/paper/1712.05812/) (2017)
- [Human irrationality: both bad and good for reward inference](https://mlsanity.com/paper/2111.06956/) (2021)
- [Cognitive science as a source of forward and inverse models of human decisions for robotics and control](https://mlsanity.com/paper/2109.00127/) (2021)
- [Generalized Hidden Parameter MDPs Transferable Model-based RL in a Handful of Trials](https://mlsanity.com/paper/2002.03072/) (2020)

## Built on by (most cited first, 4 of 4)

- [Open Ad Hoc Teamwork with Cooperative Game Theory](https://mlsanity.com/paper/2402.15259/) (2024)
- [Attaining Human`s Desirable Outcomes in Human-AI Interaction via Structural Causal Games](https://mlsanity.com/paper/2405.16588/) (2024)
- [Human-Modeling in Sequential Decision-Making: An Analysis through the Lens of Human-Aware AI](https://mlsanity.com/paper/2405.07773/) (2024)
- [Propose to Learn, Learn to Propose: Evaluability-Aware Assistance under Bounded Rationality](https://mlsanity.com/paper/2609.02242/) (2026)

## Abstract (arXiv)

We consider the problem of creating assistants that can help agents solve new sequential decision problems, assuming the agent is not able to specify the reward function explicitly to the assistant. Instead of acting in place of the agent as in current automation-based approaches, we give the assistant an advisory role and keep the agent in the loop as the main decision maker. The difficulty is that we must account for potential biases of the agent which may cause it to seemingly irrationally reject advice. To do this we introduce a novel formalization of assistance that models these biases, allowing the assistant to infer and adapt to them. We then introduce a new method for planning the assistant's actions which can scale to large decision making problems. We show experimentally that our approach adapts to these agent biases, and results in higher cumulative reward for the agent than automation-based alternatives. Lastly, we show that an approach combining advice and automation outperforms advice alone at the cost of losing some safety guarantees.