# "Give Me an Example Like This": Episodic Active Reinforcement Learning from Demonstrations

arXiv: https://arxiv.org/abs/2406.03069 · HTML: https://mlsanity.com/paper/2406.03069/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 16 of its 31 references that are in the map.
- First posted on arXiv 5 Jun 2024.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)
- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 16 of 16)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [AWAC: Accelerating Online Reinforcement Learning with Offline Datasets](https://mlsanity.com/paper/2006.09359/) (2020)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [HG-DAgger: Interactive Imitation Learning with Human Experts](https://mlsanity.com/paper/1810.02890/) (2018)
- [Parrot: Data-Driven Behavioral Priors for Reinforcement Learning](https://mlsanity.com/paper/2011.10024/) (2020)
- [Urban Driving with Conditional Imitation Learning](https://mlsanity.com/paper/1912.00177/) (2019)
- [imitation: Clean Imitation Learning Implementations](https://mlsanity.com/paper/2211.11972/) (2022)
- [Interactive Policy Learning through Confidence-Based Autonomy](https://mlsanity.com/paper/1401.3439/) (2014)
- [Improved Deep Reinforcement Learning with Expert Demonstrations for Urban Autonomous Driving](https://mlsanity.com/paper/2102.09243/) (2021)
- [Active Deep Q-learning with Demonstration](https://mlsanity.com/paper/1812.02632/) (2018)
- [A Framework for Learning from Demonstration with Minimal Human Effort](https://mlsanity.com/paper/2306.09211/) (2023)
- [Back to Reality for Imitation Learning](https://mlsanity.com/paper/2111.12867/) (2021)

## Built on by (most cited first, 2 of 2)

- [Robot Policy Transfer with Online Demonstrations: An Active Reinforcement Learning Approach](https://mlsanity.com/paper/2503.12993/) (2025)
- [Can you see how I learn? Human observers' inferences about Reinforcement Learning agents' learning processes](https://mlsanity.com/paper/2506.13583/) (2025)

## Abstract (arXiv)

Reinforcement Learning (RL) has achieved great success in sequential decision-making problems, but often at the cost of a large number of agent-environment interactions. To improve sample efficiency, methods like Reinforcement Learning from Expert Demonstrations (RLED) introduce external expert demonstrations to facilitate agent exploration during the learning process. In practice, these demonstrations, which are often collected from human users, are costly and hence often constrained to a limited amount. How to select the best set of human demonstrations that is most beneficial for learning therefore becomes a major concern. This paper presents EARLY (Episodic Active Learning from demonstration querY), an algorithm that enables a learning agent to generate optimized queries of expert demonstrations in a trajectory-based feature space. Based on a trajectory-level estimate of uncertainty in the agent's current policy, EARLY determines the optimized timing and content for feature-based queries. By querying episodic demonstrations as opposed to isolated state-action pairs, EARLY improves the human teaching experience and achieves better learning performance. We validate the effectiveness of our method in three simulated navigation tasks of increasing difficulty. The results show that our method is able to achieve expert-level performance for all three tasks with convergence over 30% faster than other baseline methods when demonstrations are generated by simulated oracle policies. The results of a follow-up pilot user study (N=18) further validate that our method can still maintain a significantly better convergence in the case of human expert demonstrators while achieving a better user experience in perceived task load and consuming significantly less human time.