On Generalization and Distributional Update for Mimicking Observations with Adequate Exploration
On the map: Offline RL and imitation learning from data → Imitation learning methods
Abstract (arXiv)
Learning from observations (LfO) replicates expert behavior without needing access to the expert's actions, making it more practical than learning from demonstrations (LfD) in many real-world scenarios. However, directly applying the on-policy training scheme in LfO worsens the sample inefficiency problem, while employing the traditional off-policy training scheme in LfO magnifies the instability issue. This paper seeks to develop an efficient and stable solution for the LfO problem. Specifically, we begin by exploring the generalization capabilities of both the reward function and policy in LfO, which provides a theoretical foundation for computation. Building on this, we modify the policy optimization method in generative adversarial imitation from observation (GAIfO) with distributional soft actor-critic (DSAC), and propose the Mimicking Observations through Distributional Update Learning with adequate Exploration (MODULE) algorithm to solve the LfO problem. MODULE incorporates the advantages of (1) high sample efficiency and training robustness enhancement in soft actor-critic (SAC), and (2) training stability in distributional reinforcement learning (RL). Extensive experiments in MuJoCo environments showcase the superior performance of MODULE over current LfO methods.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 39 of its 68 references that are in the map. First posted on arXiv 22 Jan 2025.
Builds on
39 of 68 references.
Closest ancestors · 22
Same direction, newest first
- 2024Diffusion Imitation from Observation7 Oct
- 2024A Dual Approach to Imitation Learning from Observations with Offline Datasets13 Jun
- 2023Exploring Gradient Explosion in Generative Adversarial Imitation Learning: A Probabilistic Perspective18 Dec
- 2023Imitation Learning from Observation with Automatic Discount Scheduling11 Oct
- 2023Imitation Learning from Observation through Optimal Transport2 Oct
- 2023Mimicking Better by Matching the Approximate Action Distribution16 Jun
- 2022A Ranking Game for Imitation Learning7 Feb
- 2022Adversarial Imitation Learning from Video using a State Observer1 Feb
- 2021Imitation by Predicting Observations8 Jul
- 2021Reward function shape exploration in adversarial imitation learning: an empirical study14 Apr
- 2021Off-Policy Imitation Learning from Observations25 Feb
- 2020Error Bounds of Imitating Policies and Environments22 Oct
10 more
- 2020Imitation Learning with Sinkhorn Distances20 Aug
- 2020State-only Imitation with Transition Dynamics Mismatch27 Feb
- 2020On Computation and Generalization of Generative Adversarial Imitation Learning9 Jan
- 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
- 2019Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement10 Oct
- 2019Recent Advances in Imitation Learning from Observation30 May
- 2019Provably Efficient Imitation Learning from Observation Alone27 May
- 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning9 Sep
- 2018Generative Adversarial Imitation from Observation17 Jul
- 2018Behavioral Cloning from Observation4 May
Other references · 17
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
- 2016Generative Adversarial Imitation Learningcited by 1,576
- 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
- 2017A Distributional Perspective on Reinforcement Learningcited by 743
- 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
- 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
- 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
- 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
- 2017Generalization and Equilibrium in Generative Adversarial Nets (GANs)cited by 165
- 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
- 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 87
5 more
- 2018Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learningcited by 80
- 2020Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errorscited by 80
- 2018Multi-Agent Imitation Learning for Driving Simulationcited by 49
- 2023Optimal Transport for Offline Imitation Learningcited by 42
- 2014Variance-Constrained Actor-Critic Algorithms for Discounted and Average Reward MDPscited by 28
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2022Imitation Learning by State-Only Distribution Matching
- 2021DEALIO: Data-Efficient Adversarial Learning for Imitation from Observation
- 2025Learning from Observation: A Survey of Recent Advances
- 2023MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observations
- 2019Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation
- 2024Efficient Imitation Learning with Conservative World Models
- 2021Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximation
- 2022Imitation Learning from Observations under Transition Model Disparity