# Reinforcement Learning and Inverse Reinforcement Learning with System 1 and System 2

arXiv: https://arxiv.org/abs/1811.08549 · HTML: https://mlsanity.com/paper/1811.08549/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 7 of its 49 references that are in the map.
- First posted on arXiv 19 Nov 2018.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 7 of 7)

- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [Multi-agent Reinforcement Learning in Sequential Social Dilemmas](https://mlsanity.com/paper/1702.03037/) (2017)
- [Maintaining cooperation in complex social dilemmas using deep reinforcement learning](https://mlsanity.com/paper/1707.01068/) (2017)
- [Learning the Preferences of Ignorant, Inconsistent Agents](https://mlsanity.com/paper/1512.05832/) (2015)
- [Consequentialist conditional cooperation in social dilemmas with imperfect information](https://mlsanity.com/paper/1710.06975/) (2017)
- [Time-Inconsistent Planning: A Computational Problem in Behavioral Economics](https://mlsanity.com/paper/1405.1254/) (2014)
- [Planning Problems for Sophisticated Agents with Present Bias](https://mlsanity.com/paper/1603.08177/) (2016)

## Built on by (most cited first, 4 of 4)

- [Meaningful human control: actionable properties for AI system development](https://mlsanity.com/paper/2112.01298/) (2021)
- [Designing Fiduciary Artificial Intelligence](https://mlsanity.com/paper/2308.02435/) (2023)
- [Domain-driven Metrics for Reinforcement Learning: A Case Study on Epidemic Control using Agent-based Simulation](https://mlsanity.com/paper/2508.05154/) (2025)
- [Inverse Theory of Mind Modeling for Content Recommendation: From Web Browsing to Dynamic Intelligent Interfaces](https://mlsanity.com/paper/2608.11354/) (2026)

## Abstract (arXiv)

Inferring a person's goal from their behavior is an important problem in applications of AI (e.g. automated assistants, recommender systems). The workhorse model for this task is the rational actor model - this amounts to assuming that people have stable reward functions, discount the future exponentially, and construct optimal plans. Under the rational actor assumption techniques such as inverse reinforcement learning (IRL) can be used to infer a person's goals from their actions. A competing model is the dual-system model. Here decisions are the result of an interplay between a fast, automatic, heuristic-based system 1 and a slower, deliberate, calculating system 2. We generalize the dual system framework to the case of Markov decision problems and show how to compute optimal plans for dual-system agents. We show that dual-system agents exhibit behaviors that are incompatible with rational actor assumption. We show that naive applications of rational-actor IRL to the behavior of dual-system agents can generate wrong inference about the agents' goals and suggest interventions that actually reduce the agent's overall utility. Finally, we adapt a simple IRL algorithm to correctly infer the goals of dual system decision-makers. This allows us to make interventions that help, rather than hinder, the dual-system agent's ability to reach their true goals.