# Unified Models of Human Behavioral Agents in Bandits, Contextual Bandits and RL

arXiv: https://arxiv.org/abs/2005.04544 · HTML: https://mlsanity.com/paper/2005.04544/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 May 2020.

## Built on by (most cited first, 11 of 11)

- [Deep Annotation of Therapeutic Working Alliance in Psychotherapy](https://mlsanity.com/paper/2204.05522/) (2022)
- [Neural Topic Modeling of Psychotherapy Sessions](https://mlsanity.com/paper/2204.10189/) (2022)
- [Reinforcement Learning and Bandits for Speech and Language Processing: Tutorial, Review and Outlook](https://mlsanity.com/paper/2210.13623/) (2022)
- [Working Alliance Transformer for Psychotherapy Dialogue Classification](https://mlsanity.com/paper/2210.15603/) (2022)
- [SupervisorBot: NLP-Annotated Real-Time Recommendations of Psychotherapy Treatment Strategies with Deep Reinforcement Learning](https://mlsanity.com/paper/2208.13077/) (2022)
- [Psychotherapy AI Companion with Reinforcement Learning Recommendations and Interpretable Policy Dynamics](https://mlsanity.com/paper/2303.09601/) (2023)
- [Optimal Epidemic Control as a Contextual Combinatorial Bandit with Budget](https://mlsanity.com/paper/2106.15808/) (2021)
- [Computational Inference in Cognitive Science: Operational, Societal and Ethical Considerations](https://mlsanity.com/paper/2210.13526/) (2022)
- [TherapyView: Visualizing Therapy Sessions with Temporal Topic Modeling and AI-Generated Arts](https://mlsanity.com/paper/2302.10845/) (2023)
- [Evolutionary Multi-Armed Bandits with Genetic Thompson Sampling](https://mlsanity.com/paper/2205.10113/) (2022)
- [Etat de l'art sur l'application des bandits multi-bras](https://mlsanity.com/paper/2101.00001/) (2021)

## Abstract (arXiv)

Artificial behavioral agents are often evaluated based on their consistent behaviors and performance to take sequential actions in an environment to maximize some notion of cumulative reward. However, human decision making in real life usually involves different strategies and behavioral trajectories that lead to the same empirical outcome. Motivated by clinical literature of a wide range of neurological and psychiatric disorders, we propose here a more general and flexible parametric framework for sequential decision making that involves a two-stream reward processing mechanism. We demonstrated that this framework is flexible and unified enough to incorporate a family of problems spanning multi-armed bandits (MAB), contextual bandits (CB) and reinforcement learning (RL), which decompose the sequential decision making process in different levels. Inspired by the known reward processing abnormalities of many mental disorders, our clinically-inspired agents demonstrated interesting behavioral trajectories and comparable performance on simulated tasks with particular reward distributions, a real-world dataset capturing human decision-making in gambling tasks, and the PacMan game across different reward stationarities in a lifelong learning setting.