# Inverse Delayed Reinforcement Learning

arXiv: https://arxiv.org/abs/2412.02931 · HTML: https://mlsanity.com/paper/2412.02931/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 36 of its 71 references that are in the map.
- First posted on arXiv 4 Dec 2024.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 30 of 36)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [Lipschitz regularity of deep neural networks: analysis and efficient estimation](https://mlsanity.com/paper/1805.10965/) (2018)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress](https://mlsanity.com/paper/1806.06877/) (2018)
- [Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations](https://mlsanity.com/paper/1904.06387/) (2019)
- [Learning Agile Robotic Locomotion Skills by Imitating Animals](https://mlsanity.com/paper/2004.00784/) (2020)
- [Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning](https://mlsanity.com/paper/1809.02925/) (2018)
- [Generative Adversarial Imitation from Observation](https://mlsanity.com/paper/1807.06158/) (2018)
- [Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1507.04888/) (2015)
- [RUDDER: Return Decomposition for Delayed Rewards](https://mlsanity.com/paper/1806.07857/) (2018)
- [Control of a Quadrotor with Reinforcement Learning](https://mlsanity.com/paper/1707.05110/) (2017)
- [Gradient descent GAN optimization is locally stable](https://mlsanity.com/paper/1706.04156/) (2017)
- [Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environments](https://mlsanity.com/paper/2209.15090/) (2022)
- [Scalable Bayesian Inverse Reinforcement Learning](https://mlsanity.com/paper/2102.06483/) (2021)
- [Sample-Efficient Imitation Learning via Generative Adversarial Nets](https://mlsanity.com/paper/1809.02064/) (2018)
- [Delay-Aware Model-Based Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/2005.05440/) (2020)
- [Setting up a Reinforcement Learning Task with a Real-World Robot](https://mlsanity.com/paper/1803.07067/) (2018)
- [Control Frequency Adaptation via Action Persistence in Batch Reinforcement Learning](https://mlsanity.com/paper/2002.06836/) (2020)
- [Off-Policy Reinforcement Learning with Delayed Rewards](https://mlsanity.com/paper/2106.11854/) (2021)
- [Delayed Reinforcement Learning by Imitation](https://mlsanity.com/paper/2205.05569/) (2022)
- [HIRL: Hierarchical Inverse Reinforcement Learning for Long-Horizon Tasks with Delayed Rewards](https://mlsanity.com/paper/1604.06508/) (2016)
- [Reinforcement Learning from Delayed Observations via World Models](https://mlsanity.com/paper/2403.12309/) (2024)
- [Joint Differentiable Optimization and Verification for Certified Reinforcement Learning](https://mlsanity.com/paper/2201.12243/) (2022)
- [Multi-Agent Reinforcement Learning with Reward Delays](https://mlsanity.com/paper/2212.01441/) (2022)

## Abstract (arXiv)

Inverse Reinforcement Learning (IRL) has demonstrated effectiveness in a variety of imitation tasks. In this paper, we introduce an IRL framework designed to extract rewarding features from expert trajectories affected by delayed disturbances. Instead of relying on direct observations, our approach employs an efficient off-policy adversarial training framework to derive expert features and recover optimal policies from augmented delayed observations. Empirical evaluations in the MuJoCo environment under diverse delay settings validate the effectiveness of our method. Furthermore, we provide a theoretical analysis showing that recovering expert policies from augmented delayed observations outperforms using direct delayed observations.