# Imitation Learning from Observation with Automatic Discount Scheduling

arXiv: https://arxiv.org/abs/2310.07433 · HTML: https://mlsanity.com/paper/2310.07433/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 42 of its 59 references that are in the map.
- First posted on arXiv 11 Oct 2023.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning](https://mlsanity.com/paper/2107.09645/) (2021)
- [Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design](https://mlsanity.com/paper/2012.02096/) (2020)
- [Reverse Curriculum Generation for Reinforcement Learning](https://mlsanity.com/paper/1707.05300/) (2017)
- [Generative Adversarial Imitation from Observation](https://mlsanity.com/paper/1807.06158/) (2018)
- [Meta-Gradient Reinforcement Learning](https://mlsanity.com/paper/1805.09801/) (2018)
- [Primal Wasserstein Imitation Learning](https://mlsanity.com/paper/2006.04678/) (2020)
- [Zero-Shot Visual Imitation](https://mlsanity.com/paper/1804.08606/) (2018)
- [Imitating Latent Policies from Observation](https://mlsanity.com/paper/1805.07914/) (2018)
- [Recent Advances in Imitation Learning from Observation](https://mlsanity.com/paper/1905.13566/) (2019)
- [State-Only Imitation Learning for Dexterous Manipulation](https://mlsanity.com/paper/2004.04650/) (2020)
- [Combining Self-Supervised Learning and Imitation for Vision-Based Rope Manipulation](https://mlsanity.com/paper/1703.02018/) (2017)
- [Automatic Curriculum Learning through Value Disagreement](https://mlsanity.com/paper/2006.09641/) (2020)
- [Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning](https://mlsanity.com/paper/1708.02190/) (2017)
- [Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrations](https://mlsanity.com/paper/2303.01497/) (2023)
- [Watch and Match: Supercharging Imitation with Regularized Optimal Transport](https://mlsanity.com/paper/2206.15469/) (2022)
- [Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement](https://mlsanity.com/paper/1910.04417/) (2019)
- [Discount Factor as a Regularizer in Reinforcement Learning](https://mlsanity.com/paper/2007.02040/) (2020)
- [MobILE: Model-Based Imitation Learning From Observation Alone](https://mlsanity.com/paper/2102.10769/) (2021)
- [Optimal Transport for Offline Imitation Learning](https://mlsanity.com/paper/2303.13971/) (2023)

## Built on by (most cited first, 10 of 10)

- [Robot Policy Learning with Temporal Optimal Transport Reward](https://mlsanity.com/paper/2410.21795/) (2024)
- [Imitation Learning from a Single Temporally Misaligned Video](https://mlsanity.com/paper/2502.05397/) (2025)
- [TimeRewarder: Learning Dense Reward from Passive Videos via Frame-wise Temporal Distance](https://mlsanity.com/paper/2509.26627/) (2025)
- [On Generalization and Distributional Update for Mimicking Observations with Adequate Exploration](https://mlsanity.com/paper/2501.12785/) (2025)
- [Minimal Ingredients for Reward Assignment from Expert Demonstrations](https://mlsanity.com/paper/2506.06793/) (2025)
- [Search-Based Credit Assignment for Offline Preference-Based Reinforcement Learning](https://mlsanity.com/paper/2508.15327/) (2025)
- [Learning from Observation: A Survey of Recent Advances](https://mlsanity.com/paper/2509.19379/) (2025)
- [Imitation Learning in the Deep Learning Era: A Novel Taxonomy and Recent Advances](https://mlsanity.com/paper/2511.03565/) (2025)
- [Going Beyond Expert Performance via Deep Implicit Imitation Reinforcement Learning](https://mlsanity.com/paper/2511.03616/) (2025)
- [OPRIDE: Offline Preference-based Reinforcement Learning via In-Dataset Exploration](https://mlsanity.com/paper/2604.02349/) (2026)

## Abstract (arXiv)

Humans often acquire new skills through observation and imitation. For robotic agents, learning from the plethora of unlabeled video demonstration data available on the Internet necessitates imitating the expert without access to its action, presenting a challenge known as Imitation Learning from Observations (ILfO). A common approach to tackle ILfO problems is to convert them into inverse reinforcement learning problems, utilizing a proxy reward computed from the agent's and the expert's observations. Nonetheless, we identify that tasks characterized by a progress dependency property pose significant challenges for such approaches; in these tasks, the agent needs to initially learn the expert's preceding behaviors before mastering the subsequent ones. Our investigation reveals that the main cause is that the reward signals assigned to later steps hinder the learning of initial behaviors. To address this challenge, we present a novel ILfO framework that enables the agent to master earlier behaviors before advancing to later ones. We introduce an Automatic Discount Scheduling (ADS) mechanism that adaptively alters the discount factor in reinforcement learning during the training phase, prioritizing earlier rewards initially and gradually engaging later rewards only when the earlier behaviors have been mastered. Our experiments, conducted on nine Meta-World tasks, demonstrate that our method significantly outperforms state-of-the-art methods across all tasks, including those that are unsolvable by them.