# Model-Based Imitation Learning Using Entropy Regularization of Model and Policy

arXiv: https://arxiv.org/abs/2206.10101 · HTML: https://mlsanity.com/paper/2206.10101/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 23 of its 40 references that are in the map.
- First posted on arXiv 21 Jun 2022.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 23 of 23)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress](https://mlsanity.com/paper/1806.06877/) (2018)
- [Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning](https://mlsanity.com/paper/1809.02925/) (2018)
- [Imitation Learning via Off-Policy Distribution Matching](https://mlsanity.com/paper/1912.05032/) (2019)
- [A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models](https://mlsanity.com/paper/1611.03852/) (2016)
- [From Variational to Deterministic Autoencoders](https://mlsanity.com/paper/1903.12436/) (2019)
- [Off-Policy Imitation Learning from Observations](https://mlsanity.com/paper/2102.13185/) (2021)
- [Sample-Efficient Imitation Learning via Generative Adversarial Nets](https://mlsanity.com/paper/1809.02064/) (2018)
- [SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards](https://mlsanity.com/paper/1905.11108/) (2019)
- [Inverse Reinforcement Learning with Simultaneous Estimation of Rewards and Dynamics](https://mlsanity.com/paper/1604.03912/) (2016)
- [Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization](https://mlsanity.com/paper/2006.13258/) (2020)
- [OPIRL: Sample Efficient Off-Policy Inverse Reinforcement Learning via Distribution Matching](https://mlsanity.com/paper/2109.04307/) (2021)
- [When Will Generative Adversarial Imitation Learning Algorithms Attain Global Convergence](https://mlsanity.com/paper/2006.13506/) (2020)
- [Generative Adversarial Imitation Learning with Neural Networks: Global Optimality and Convergence Rate](https://mlsanity.com/paper/2003.03709/) (2020)
- [Forward and inverse reinforcement learning sharing network weights and hyperparameters](https://mlsanity.com/paper/2008.07284/) (2020)
- [DEALIO: Data-Efficient Adversarial Learning for Imitation from Observation](https://mlsanity.com/paper/2104.00163/) (2021)
- [Discriminator Soft Actor Critic without Extrinsic Rewards](https://mlsanity.com/paper/2001.06808/) (2020)
- [Dyna-AIL : Adversarial Imitation Learning by Planning](https://mlsanity.com/paper/1903.03234/) (2019)

## Built on by (most cited first, 1 of 1)

- [Statistical analysis of Inverse Entropy-regularized Reinforcement Learning](https://mlsanity.com/paper/2512.06956/) (2025)

## Abstract (arXiv)

Approaches based on generative adversarial networks for imitation learning are promising because they are sample efficient in terms of expert demonstrations. However, training a generator requires many interactions with the actual environment because model-free reinforcement learning is adopted to update a policy. To improve the sample efficiency using model-based reinforcement learning, we propose model-based Entropy-Regularized Imitation Learning (MB-ERIL) under the entropy-regularized Markov decision process to reduce the number of interactions with the actual environment. MB-ERIL uses two discriminators. A policy discriminator distinguishes the actions generated by a robot from expert ones, and a model discriminator distinguishes the counterfactual state transitions generated by the model from the actual ones. We derive structured discriminators so that the learning of the policy and the model is efficient. Computer simulations and real robot experiments show that MB-ERIL achieves a competitive performance and significantly improves the sample efficiency compared to baseline methods.