# A Differential Dynamic Programming Framework for Inverse Reinforcement Learning

arXiv: https://arxiv.org/abs/2407.19902 · HTML: https://mlsanity.com/paper/2407.19902/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 12 of its 40 references that are in the map.
- First posted on arXiv 29 Jul 2024.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)
- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 12 of 12)

- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Approximation Methods for Bilevel Programming](https://mlsanity.com/paper/1802.02246/) (2018)
- [Recent Advances in Reinforcement Learning in Finance](https://mlsanity.com/paper/2112.04553/) (2021)
- [Correcting Robot Plans with Natural Language Feedback](https://mlsanity.com/paper/2204.05186/) (2022)
- [Pontryagin Differentiable Programming: An End-to-End Learning and Control Framework](https://mlsanity.com/paper/1912.12970/) (2019)
- [Safe Pontryagin Differentiable Programming](https://mlsanity.com/paper/2105.14937/) (2021)
- [Combining Physics and Deep Learning to learn Continuous-Time Dynamics Models](https://mlsanity.com/paper/2110.01894/) (2021)
- [Learning from Sparse Demonstrations](https://mlsanity.com/paper/2008.02159/) (2020)
- [Online Inverse Optimal Control for Control-Constrained Discrete-Time Systems on Finite and Infinite Horizons](https://mlsanity.com/paper/2005.06153/) (2020)
- [Inverse Optimal Control from Incomplete Trajectory Observations](https://mlsanity.com/paper/1803.07696/) (2018)
- [Interior Point Differential Dynamic Programming](https://mlsanity.com/paper/2004.12710/) (2020)
- [Constrained Differential Dynamic Programming Revisited](https://mlsanity.com/paper/2005.00985/) (2020)

## Built on by (most cited first, 2 of 2)

- [DiffCoord: Differentiable Coordination for Distributed Multi-Agent Trajectory Optimization](https://mlsanity.com/paper/2509.01630/) (2025)
- [Adaptive Inverse Reinforcement Learning with Online Off-Policy Data Collection](https://mlsanity.com/paper/2511.15171/) (2025)

## Abstract (arXiv)

A differential dynamic programming (DDP)-based framework for inverse reinforcement learning (IRL) is introduced to recover the parameters in the cost function, system dynamics, and constraints from demonstrations. Different from existing work, where DDP was used for the inner forward problem with inequality constraints, our proposed framework uses it for efficient computation of the gradient required in the outer inverse problem with equality and inequality constraints. The equivalence between the proposed method and existing methods based on Pontryagin's Maximum Principle (PMP) is established. More importantly, using this DDP-based IRL with an open-loop loss function, a closed-loop IRL framework is presented. In this framework, a loss function is proposed to capture the closed-loop nature of demonstrations. It is shown to be better than the commonly used open-loop loss function. We show that the closed-loop IRL framework reduces to a constrained inverse optimal control problem under certain assumptions. Under these assumptions and a rank condition, it is proven that the learning parameters can be recovered from the demonstration data. The proposed framework is extensively evaluated through four numerical robot examples and one real-world quadrotor system. The experiments validate the theoretical results and illustrate the practical relevance of the approach.