# Reinforcement learning with Demonstrations from Mismatched Task under Sparse Reward

arXiv: https://arxiv.org/abs/2212.01509 · HTML: https://mlsanity.com/paper/2212.01509/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 39 references that are in the map.
- First posted on arXiv 3 Dec 2022.

## Builds on (most cited first, 26 of 26)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations](https://mlsanity.com/paper/1904.06387/) (2019)
- [Generative Adversarial Imitation from Observation](https://mlsanity.com/paper/1807.06158/) (2018)
- [Learning by Playing - Solving Sparse Reward Tasks from Scratch](https://mlsanity.com/paper/1802.10567/) (2018)
- [State-Only Imitation Learning for Dexterous Manipulation](https://mlsanity.com/paper/2004.04650/) (2020)
- [Learning from Suboptimal Demonstration via Self-Supervised Reward Regression](https://mlsanity.com/paper/2010.11723/) (2020)
- [Provably Efficient Imitation Learning from Observation Alone](https://mlsanity.com/paper/1905.10948/) (2019)
- [Reinforcement Learning from Imperfect Demonstrations](https://mlsanity.com/paper/1802.05313/) (2018)
- [State Alignment-based Imitation Learning](https://mlsanity.com/paper/1911.10947/) (2019)
- [Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstration](https://mlsanity.com/paper/2202.04628/) (2022)
- [Offline Meta-Reinforcement Learning with Online Self-Supervision](https://mlsanity.com/paper/2107.03974/) (2021)
- [Heuristic-Guided Reinforcement Learning](https://mlsanity.com/paper/2106.02757/) (2021)
- [Offline Meta-Reinforcement Learning for Industrial Insertion](https://mlsanity.com/paper/2110.04276/) (2021)
- [State-only Imitation with Transition Dynamics Mismatch](https://mlsanity.com/paper/2002.11879/) (2020)
- [Learning from Imperfect Demonstrations from Agents with Varying Dynamics](https://mlsanity.com/paper/2103.05910/) (2021)
- [Self-Imitation Learning by Planning](https://mlsanity.com/paper/2103.13834/) (2021)
- [Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models](https://mlsanity.com/paper/2011.01298/) (2020)
- [Learning Feasibility to Imitate Demonstrators with Different Dynamics](https://mlsanity.com/paper/2110.15142/) (2021)

## Built on by (most cited first, 4 of 4)

- [ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning](https://mlsanity.com/paper/2505.22094/) (2025)
- [Fast Ergodic Search with Kernel Functions](https://mlsanity.com/paper/2403.01536/) (2024)
- [TW-CRL: Time-Weighted Contrastive Reward Learning for Efficient Inverse Reinforcement Learning](https://mlsanity.com/paper/2504.05585/) (2025)
- [Intrinsic Vicarious Conditioning for Deep Reinforcement Learning](https://mlsanity.com/paper/2605.12224/) (2026)

## Abstract (arXiv)

Reinforcement learning often suffer from the sparse reward issue in real-world robotics problems. Learning from demonstration (LfD) is an effective way to eliminate this problem, which leverages collected expert data to aid online learning. Prior works often assume that the learning agent and the expert aim to accomplish the same task, which requires collecting new data for every new task. In this paper, we consider the case where the target task is mismatched from but similar with that of the expert. Such setting can be challenging and we found existing LfD methods can not effectively guide learning in mismatched new tasks with sparse rewards. We propose conservative reward shaping from demonstration (CRSfD), which shapes the sparse rewards using estimated expert value function. To accelerate learning processes, CRSfD guides the agent to conservatively explore around demonstrations. Experimental results of robot manipulation tasks show that our approach outperforms baseline LfD methods when transferring demonstrations collected in a single task to other different but similar tasks.