# Reinforced Imitation in Heterogeneous Action Space

arXiv: https://arxiv.org/abs/1904.03438 · HTML: https://mlsanity.com/paper/1904.03438/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- First posted on arXiv 6 Apr 2019.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Built on by (most cited first, 9 of 9)

- [Offline Learning from Demonstrations and Unlabeled Experience](https://mlsanity.com/paper/2011.13885/) (2020)
- [Task-Relevant Adversarial Imitation Learning](https://mlsanity.com/paper/1910.01077/) (2019)
- [Positive-Unlabeled Reward Learning](https://mlsanity.com/paper/1911.00459/) (2019)
- [Imitation Learning from Observations under Transition Model Disparity](https://mlsanity.com/paper/2204.11446/) (2022)
- [Collaborative Training of Heterogeneous Reinforcement Learning Agents in Environments with Sparse Rewards: What and When to Share?](https://mlsanity.com/paper/2202.12174/) (2022)
- [Self-Imitation Learning from Demonstrations](https://mlsanity.com/paper/2203.10905/) (2022)
- [Combining RL and IL using a dynamic, performance-based modulation over learning signals and its application to local planning](https://mlsanity.com/paper/2405.09760/) (2024)
- [D-Shape: Demonstration-Shaped Reinforcement Learning via Goal Conditioning](https://mlsanity.com/paper/2210.14428/) (2022)
- [Transfer Reinforcement Learning in Heterogeneous Action Spaces using Subgoal Mapping](https://mlsanity.com/paper/2410.14484/) (2024)

## Abstract (arXiv)

Imitation learning is an effective alternative approach to learn a policy when the reward function is sparse. In this paper, we consider a challenging setting where an agent and an expert use different actions from each other. We assume that the agent has access to a sparse reward function and state-only expert observations. We propose a method which gradually balances between the imitation learning cost and the reinforcement learning objective. In addition, this method adapts the agent's policy based on either mimicking expert behavior or maximizing sparse reward. We show, through navigation scenarios, that (i) an agent is able to efficiently leverage sparse rewards to outperform standard state-only imitation learning, (ii) it can learn a policy even when its actions are different from the expert, and (iii) the performance of the agent is not bounded by that of the expert, due to the optimized usage of sparse rewards.