# Combining imitation and deep reinforcement learning to accomplish human-level performance on a virtual foraging task

arXiv: https://arxiv.org/abs/2203.06250 · HTML: https://mlsanity.com/paper/2203.06250/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 64 references that are in the map.
- First posted on arXiv 11 Mar 2022.

## Builds on (most cited first, 24 of 24)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [An Algorithmic Perspective on Imitation Learning](https://mlsanity.com/paper/1811.06711/) (2018)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [Challenges of Real-World Reinforcement Learning](https://mlsanity.com/paper/1904.12901/) (2019)
- [A Divergence Minimization Perspective on Imitation Learning Methods](https://mlsanity.com/paper/1911.02256/) (2019)
- [Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO](https://mlsanity.com/paper/2005.12729/) (2020)
- [What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study](https://mlsanity.com/paper/2006.05990/) (2020)
- [Reinforcement and Imitation Learning via Interactive No-Regret Learning](https://mlsanity.com/paper/1406.5979/) (2014)
- [Generalized Proximal Policy Optimization with Sample Reuse](https://mlsanity.com/paper/2111.00072/) (2021)
- [Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learning](https://mlsanity.com/paper/1805.11240/) (2018)
- [Fast Policy Learning through Imitation and Reinforcement](https://mlsanity.com/paper/1805.10413/) (2018)
- [Guided Exploration with Proximal Policy Optimization using a Single Demonstration](https://mlsanity.com/paper/2007.03328/) (2020)
- [Forward and inverse reinforcement learning sharing network weights and hyperparameters](https://mlsanity.com/paper/2008.07284/) (2020)
- [Uncertainty-Aware Policy Optimization: A Robust, Adaptive Trust Region Approach](https://mlsanity.com/paper/2012.10791/) (2020)

## Built on by (most cited first, 4 of 4)

- [Adversarial Imitation Learning from Visual Observations using Latent Information](https://mlsanity.com/paper/2309.17371/) (2023)
- [Reinforcement Learning for Active Matter](https://mlsanity.com/paper/2503.23308/) (2025)
- [A Model-Based Approach for Improving Reinforcement Learning Efficiency Leveraging Expert Observations](https://mlsanity.com/paper/2402.18836/) (2024)
- [Physics-informed Goal-Conditioned Reinforcement Learning under Hybrid Contact Dynamics](https://mlsanity.com/paper/2605.30503/) (2026)

## Abstract (arXiv)

We develop a simple framework to learn bio-inspired foraging policies using human data. We conduct an experiment where humans are virtually immersed in an open field foraging environment and are trained to collect the highest amount of rewards. A Markov Decision Process (MDP) framework is introduced to model the human decision dynamics. Then, Imitation Learning (IL) based on maximum likelihood estimation is used to train Neural Networks (NN) that map human decisions to observed states. The results show that passive imitation substantially underperforms humans. We further refine the human-inspired policies via Reinforcement Learning (RL) using the on-policy Proximal Policy Optimization (PPO) algorithm which shows better stability than other algorithms and can steadily improve the policies pretrained with IL. We show that the combination of IL and RL can match human results and that good performance strongly depends on combining the allocentric information with an egocentric representation of the environment.