# Inverse Reinforcement Learning via Convex Optimization

arXiv: https://arxiv.org/abs/2501.15957 · HTML: https://mlsanity.com/paper/2501.15957/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 5 of its 16 references that are in the map.
- First posted on arXiv 27 Jan 2025.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)
- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 5 of 5)

- [CVXPY: A Python-Embedded Modeling Language for Convex Optimization](https://mlsanity.com/paper/1603.00943/) (2016)
- [A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress](https://mlsanity.com/paper/1806.06877/) (2018)
- [Chain-of-Thought Predictive Control](https://mlsanity.com/paper/2304.00776/) (2023)
- [Inverse Rational Control with Partially Observable Continuous Nonlinear Dynamics](https://mlsanity.com/paper/1908.04696/) (2019)
- [Multi-intention Inverse Q-learning for Interpretable Behavior Representation](https://mlsanity.com/paper/2311.13870/) (2023)

## Abstract (arXiv)

We consider the inverse reinforcement learning (IRL) problem, where an unknown reward function of some Markov decision process is estimated based on observed expert demonstrations. In most existing approaches, IRL is formulated and solved as a nonconvex optimization problem, posing challenges in scenarios where robustness and reproducibility are critical. We discuss a convex formulation of the IRL problem (CIRL) initially proposed by Ng and Russel, and reformulate the problem such that the domain-specific language CVXPY can be applied directly to specify and solve the convex problem. We also extend the CIRL problem to scenarios where the expert policy is not given analytically but by trajectory as state-action pairs, which can be strongly inconsistent with optimality, by augmenting some of the constraints. Theoretical analysis and practical implementation for hyperparameter auto-selection are introduced. This note helps the users to easily apply CIRL for their problems, without background knowledge on convex optimization.