# EvIL: Evolution Strategies for Generalisable Imitation Learning

arXiv: https://arxiv.org/abs/2406.11905 · HTML: https://mlsanity.com/paper/2406.11905/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 29 of its 60 references that are in the map.
- First posted on arXiv 15 Jun 2024.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 29 of 29)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [MOReL : Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2005.05951/) (2020)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation](https://mlsanity.com/paper/2106.13281/) (2021)
- [Human-level performance in first-person multiplayer games with population-based deep reinforcement learning](https://mlsanity.com/paper/1807.01281/) (2018)
- [A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models](https://mlsanity.com/paper/1611.03852/) (2016)
- [Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research](https://mlsanity.com/paper/2310.08710/) (2023)
- [Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning](https://mlsanity.com/paper/1712.06567/) (2017)
- [Discovered Policy Optimisation](https://mlsanity.com/paper/2210.05639/) (2022)
- [Evolved Policy Gradients](https://mlsanity.com/paper/1802.04821/) (2018)
- [Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap](https://mlsanity.com/paper/2103.03236/) (2021)
- [Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real world](https://mlsanity.com/paper/2206.09889/) (2022)
- [Model-Free Opponent Shaping](https://mlsanity.com/paper/2205.01447/) (2022)
- [evosax: JAX-based Evolution Strategies](https://mlsanity.com/paper/2212.04180/) (2022)
- [Hybrid Inverse Reinforcement Learning](https://mlsanity.com/paper/2402.08848/) (2024)
- [Bridging RL Theory and Practice with the Effective Horizon](https://mlsanity.com/paper/2304.09853/) (2023)
- [Minimax Optimal Online Imitation Learning via Replay Estimation](https://mlsanity.com/paper/2205.15397/) (2022)
- [Discovering Temporally-Aware Reinforcement Learning Algorithms](https://mlsanity.com/paper/2402.05828/) (2024)
- [Adversarial Cheap Talk](https://mlsanity.com/paper/2211.11030/) (2022)
- [A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learning](https://mlsanity.com/paper/2112.15400/) (2021)
- [Scaling Opponent Shaping to High Dimensional Games](https://mlsanity.com/paper/2312.12568/) (2023)
- [Massively Scalable Inverse Reinforcement Learning in Google Maps](https://mlsanity.com/paper/2305.11290/) (2023)
- [Fast Population-Based Reinforcement Learning on a Single Machine](https://mlsanity.com/paper/2206.08888/) (2022)
- [Toward Computationally Efficient Inverse Reinforcement Learning via Reward Shaping](https://mlsanity.com/paper/2312.09983/) (2023)

## Built on by (most cited first, 8 of 8)

- [Diffusing States and Matching Scores: A New Framework for Imitation Learning](https://mlsanity.com/paper/2410.13855/) (2024)
- [JaxLife: An Open-Ended Agentic Simulator](https://mlsanity.com/paper/2409.00853/) (2024)
- [Efficient Imitation under Misspecification](https://mlsanity.com/paper/2503.13162/) (2025)
- [Your Learned Constraint is Secretly a Backward Reachable Tube](https://mlsanity.com/paper/2501.15618/) (2025)
- [EvoRL: A GPU-accelerated Framework for Evolutionary Reinforcement Learning](https://mlsanity.com/paper/2501.15129/) (2025)
- [On Discovering Algorithms for Adversarial Imitation Learning](https://mlsanity.com/paper/2510.00922/) (2025)
- [Celo2: Towards Learned Optimization Free Lunch](https://mlsanity.com/paper/2602.19142/) (2026)
- [Project Auto-World: Towards Automated Benchmarking of Neural Relational Reasoners](https://mlsanity.com/paper/2606.24965/) (2026)

## Abstract (arXiv)

Often times in imitation learning (IL), the environment we collect expert demonstrations in and the environment we want to deploy our learned policy in aren't exactly the same (e.g. demonstrations collected in simulation but deployment in the real world). Compared to policy-centric approaches to IL like behavioural cloning, reward-centric approaches like inverse reinforcement learning (IRL) often better replicate expert behaviour in new environments. This transfer is usually performed by optimising the recovered reward under the dynamics of the target environment. However, (a) we find that modern deep IL algorithms frequently recover rewards which induce policies far weaker than the expert, even in the same environment the demonstrations were collected in. Furthermore, (b) these rewards are often quite poorly shaped, necessitating extensive environment interaction to optimise effectively. We provide simple and scalable fixes to both of these concerns. For (a), we find that reward model ensembles combined with a slightly different training objective significantly improves re-training and transfer performance. For (b), we propose a novel evolution-strategies based method EvIL to optimise for a reward-shaping term that speeds up re-training in the target environment, closing a gap left open by the classical theory of IRL. On a suite of continuous control tasks, we are able to re-train policies in target (and source) environments more interaction-efficiently than prior work.