# Fitting Reinforcement Learning Model to Behavioral Data under Bandits

arXiv: https://arxiv.org/abs/2511.04454 · HTML: https://mlsanity.com/paper/2511.04454/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 16 of its 67 references that are in the map.
- First posted on arXiv 6 Nov 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 16 of 16)

- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [CVXPY: A Python-Embedded Modeling Language for Convex Optimization](https://mlsanity.com/paper/1603.00943/) (2016)
- [Introduction to Multi-Armed Bandits](https://mlsanity.com/paper/1904.07272/) (2019)
- [A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress](https://mlsanity.com/paper/1806.06877/) (2018)
- [A Rewriting System for Convex Optimization Problems](https://mlsanity.com/paper/1709.04494/) (2017)
- [Inverse Contextual Bandits: Learning How Behavior Evolves over Time](https://mlsanity.com/paper/2107.06317/) (2021)
- [Multi-intention Inverse Q-learning for Interpretable Behavior Representation](https://mlsanity.com/paper/2311.13870/) (2023)
- [Disciplined Multi-Convex Programming](https://mlsanity.com/paper/1609.03285/) (2016)
- [Model-Based Derivative-Free Optimization Methods and Software](https://mlsanity.com/paper/2210.12018/) (2022)
- [Inverse Reinforcement Learning with Switching Rewards and History Dependency for Characterizing Animal Behaviors](https://mlsanity.com/paper/2501.12633/) (2025)
- [Learning Reward Machines from Partially Observed Policies](https://mlsanity.com/paper/2502.03762/) (2025)
- [Deep multi-intentional inverse reinforcement learning for cognitive multi-function radar inverse cognition](https://mlsanity.com/paper/2408.08478/) (2024)
- [Solving Inverse Problem for Multi-armed Bandits via Convex Optimization](https://mlsanity.com/paper/2501.18945/) (2025)
- [Disciplined Biconvex Programming](https://mlsanity.com/paper/2511.01813/) (2025)
- [NeuRL: Closed-form Inverse Reinforcement Learning for Neural Decoding](https://mlsanity.com/paper/2204.04733/) (2022)
- [Learning Task-Agnostic Motifs to Capture the Continuous Nature of Animal Behavior](https://mlsanity.com/paper/2506.15190/) (2025)

## Abstract (arXiv)

We consider the problem of fitting a reinforcement learning (RL) model to some given behavioral data under a multi-armed bandit environment. These models have received much attention in recent years for characterizing human and animal decision making behavior. We provide a generic mathematical optimization problem formulation for the fitting problem of a wide range of RL models that appear frequently in scientific research applications. We then provide a detailed theoretical analysis of its convexity properties. Based on the theoretical results, we introduce a novel solution method for the fitting problem of RL models based on convex relaxation and optimization. Our method is then evaluated in several simulated and real-world bandit environments to compare with some benchmark methods that appear in the literature. Numerical results indicate that our method achieves comparable performance to the state-of-the-art, while significantly reducing computation time. We also provide an open-source Python package for our proposed method to empower researchers to apply it in the analysis of their datasets directly, without prior knowledge of convex optimization.