# Forward and inverse reinforcement learning sharing network weights and hyperparameters

arXiv: https://arxiv.org/abs/2008.07284 · HTML: https://mlsanity.com/paper/2008.07284/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 31 of its 75 references that are in the map.
- First posted on arXiv 17 Aug 2020.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [Improving Sample Efficiency in Model-Free Reinforcement Learning from Images](https://mlsanity.com/paper/1910.01741/) (2019)
- [Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning](https://mlsanity.com/paper/1809.02925/) (2018)
- [DART: Noise Injection for Robust Imitation Learning](https://mlsanity.com/paper/1703.09327/) (2017)
- [A Divergence Minimization Perspective on Imitation Learning Methods](https://mlsanity.com/paper/1911.02256/) (2019)
- [Generative Adversarial Imitation from Observation](https://mlsanity.com/paper/1807.06158/) (2018)
- [Understanding the impact of entropy on policy optimization](https://mlsanity.com/paper/1811.11214/) (2018)
- [From Variational to Deterministic Autoencoders](https://mlsanity.com/paper/1903.12436/) (2019)
- [Imitation Learning as f-Divergence Minimization](https://mlsanity.com/paper/1905.12888/) (2019)
- [Discount Factor as a Regularizer in Reinforcement Learning](https://mlsanity.com/paper/2007.02040/) (2020)
- [OptionGAN: Learning Joint Reward-Policy Options using Generative Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1709.06683/) (2017)
- [Sample-Efficient Imitation Learning via Generative Adversarial Nets](https://mlsanity.com/paper/1809.02064/) (2018)
- [SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards](https://mlsanity.com/paper/1905.11108/) (2019)
- [Adversarial Imitation Learning from Incomplete Demonstrations](https://mlsanity.com/paper/1905.12310/) (2019)
- [Entropic Regularization of Markov Decision Processes](https://mlsanity.com/paper/1907.04214/) (2019)
- [TD-Regularized Actor-Critic Methods](https://mlsanity.com/paper/1812.08288/) (2018)
- [Online Meta-learning by Parallel Algorithm Competition](https://mlsanity.com/paper/1702.07490/) (2017)
- [Integration of Imitation Learning using GAIL and Reinforcement Learning using Task-achievement Rewards via Probabilistic Graphical Model](https://mlsanity.com/paper/1907.02140/) (2019)
- [Situated GAIL: Multitask imitation using task-conditioned adversarial inverse reinforcement learning](https://mlsanity.com/paper/1911.00238/) (2019)

## Built on by (most cited first, 6 of 6)

- [Combining imitation and deep reinforcement learning to accomplish human-level performance on a virtual foraging task](https://mlsanity.com/paper/2203.06250/) (2022)
- [Model-Based Imitation Learning Using Entropy Regularization of Model and Policy](https://mlsanity.com/paper/2206.10101/) (2022)
- [Data-Driven Inverse Optimal Control for Continuous-Time Nonlinear Systems](https://mlsanity.com/paper/2503.09090/) (2025)
- [Cautious Policy Programming: Exploiting KL Regularization in Monotonic Policy Improvement for Reinforcement Learning](https://mlsanity.com/paper/2107.05798/) (2021)
- [Choosy Babies Need One Coach: Inducing Mode-Seeking Behavior in BabyLlama with Reverse KL Divergence](https://mlsanity.com/paper/2410.22081/) (2024)
- [Adaptive Inverse Reinforcement Learning with Online Off-Policy Data Collection](https://mlsanity.com/paper/2511.15171/) (2025)

## Abstract (arXiv)

This paper proposes model-free imitation learning named Entropy-Regularized Imitation Learning (ERIL) that minimizes the reverse Kullback-Leibler (KL) divergence. ERIL combines forward and inverse reinforcement learning (RL) under the framework of an entropy-regularized Markov decision process. An inverse RL step computes the log-ratio between two distributions by evaluating two binary discriminators. The first discriminator distinguishes the state generated by the forward RL step from the expert's state. The second discriminator, which is structured by the theory of entropy regularization, distinguishes the state-action-next-state tuples generated by the learner from the expert ones. One notable feature is that the second discriminator shares hyperparameters with the forward RL, which can be used to control the discriminator's ability. A forward RL step minimizes the reverse KL estimated by the inverse RL step. We show that minimizing the reverse KL divergence is equivalent to finding an optimal policy. Our experimental results on MuJoCo-simulated environments and vision-based reaching tasks with a robotic arm show that ERIL is more sample-efficient than the baseline methods. We apply the method to human behaviors that perform a pole-balancing task and describe how the estimated reward functions show how every subject achieves her goal.