mlsanityData to 14 Oct 2026

Fitting Reinforcement Learning Model to Behavioral Data under Bandits

arXiv 2511.04454 ↗First posted 6 Nov 2025cs.CE cs.LG math.OCcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

We consider the problem of fitting a reinforcement learning (RL) model to some given behavioral data under a multi-armed bandit environment. These models have received much attention in recent years for characterizing human and animal decision making behavior. We provide a generic mathematical optimization problem formulation for the fitting problem of a wide range of RL models that appear frequently in scientific research applications. We then provide a detailed theoretical analysis of its convexity properties. Based on the theoretical results, we introduce a novel solution method for the fitting problem of RL models based on convex relaxation and optimization. Our method is then evaluated in several simulated and real-world bandit environments to compare with some benchmark methods that appear in the literature. Numerical results indicate that our method achieves comparable performance to the state-of-the-art, while significantly reducing computation time. We also provide an open-source Python package for our proposed method to empower researchers to apply it in the analysis of their datasets directly, without prior knowledge of convex optimization.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 16 of its 67 references that are in the map. First posted on arXiv 6 Nov 2025.

Builds on

16 of 67 references.

References · 16

  1. 2021Convex optimizationcited by 2,515
  2. 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 641
  3. 2019Introduction to Multi-Armed Banditscited by 481
  4. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 298
  5. 2017A Rewriting System for Convex Optimization Problemscited by 179
  6. 2021Inverse Contextual Bandits: Learning How Behavior Evolves over Timecited by 12
  7. 2023Multi-intention Inverse Q-learning for Interpretable Behavior Representationcited by 9
  8. 2016Disciplined Multi-Convex Programmingcited by 6
  9. 2022Model-Based Derivative-Free Optimization Methods and Softwarecited by 6
  10. 2025Inverse Reinforcement Learning with Switching Rewards and History Dependency for Characterizing Animal Behaviorscited by 6
  11. 2025Learning Reward Machines from Partially Observed Policiescited by 3
  12. 2024Deep multi-intentional inverse reinforcement learning for cognitive multi-function radar inverse cognitioncited by 2
4 more
  1. 2025Solving Inverse Problem for Multi-armed Bandits via Convex Optimizationcited by 2
  2. 2025Disciplined Biconvex Programmingcited by 2
  3. 2022NeuRL: Closed-form Inverse Reinforcement Learning for Neural Decodingcited by 1
  4. 2025Learning Task-Agnostic Motifs to Capture the Continuous Nature of Animal Behaviorcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025One Framework to Rule Them All: Unifying RL-Based and RL-Free Methods in RLHF
  2. 2025Statistical and Algorithmic Foundations of Reinforcement Learning
  3. 2022Oracle Inequalities for Model Selection in Offline Reinforcement Learning
  4. 2026Mathematical methods of reinforcement learning
  5. 2024Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Review
  6. 2025Reinforcement Learning from Human Feedback
  7. 2025Behavior Preference Regression for Offline Reinforcement Learning
  8. 2025Reward Models in Deep Reinforcement Learning: A Survey