mlsanityData to 14 Oct 2026

Adaptive Inverse Reinforcement Learning with Online Off-Policy Data Collection

arXiv 2511.15171 ↗First posted 19 Nov 2025math.OCcited by 0

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation

Abstract (arXiv)

In this paper, the inverse reinforcement learning (IRL) problem is addressed to reconstruct the unknown cost function underlying an observed optimal policy in a model-free manner, whose online adaptation with completely off-policy system data still remains unclear in the literature. Without prior knowledge of the system model parameters, an adaptive and direct learning rule for the cost parameter is proposed using online off-policy system data, which only needs to satisfy the mild persistently exciting condition in the general data-driven paradigm. The adaptive and online IRL algorithm is achieved by designing full Nesterov-Todd (NT)-step primal-dual interior-point iterations. Despite solving a nonlinear and time-varying semi-definite program (SDP), the influence of system noise is rigorously analyzed, and the proposed online algorithm is shown to achieve a sublinear convergence. The proposed method is further generalized to nonlinear IRL based on differential dynamic programming. The gradient of the loss function is directly obtained via a backward pass, which eliminates the need to repeatedly solve forward RL problems as in conventional bi-level IRL frameworks. Finally, the efficiency and effectiveness of the proposed algorithms are demonstrated by numerical examples.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 8 of its 31 references that are in the map. First posted on arXiv 19 Nov 2025.

Builds on

8 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024A Differential Dynamic Programming Framework for Inverse Reinforcement Learning29 Jul

Other references · 7

  1. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  2. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
  3. 2019Pontryagin Differentiable Programming: An End-to-End Learning and Control Frameworkcited by 52
  4. 2021Efficient Off-Policy Q-Learning for Data-Based Discrete-Time LQR Problemscited by 9
  5. 2024Data-Enabled Policy Optimization for Direct Adaptive Learning of the LQRcited by 9
  6. 2020Forward and inverse reinforcement learning sharing network weights and hyperparameterscited by 6
  7. 2018Continuous-Time Inverse Quadratic Optimal Control Problemcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2018Online inverse reinforcement learning for nonlinear systems
  2. 2024Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms
  3. 2025Offline Reinforcement Learning via Inverse Optimization
  4. 2018A Framework and Method for Online Inverse Reinforcement Learning
  5. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
  6. 2020Online inverse reinforcement learning with limited data
  7. 2024Inverse reinforcement learning by expert imitation for the stochastic linear-quadratic optimal control problem
  8. 2025Data-Driven Inverse Optimal Control for Continuous-Time Nonlinear Systems