Enhancing Online Reinforcement Learning with Meta-Learned Objective from Offline Data
On the map: Offline RL and imitation learning from data → Imitation learning methods
Abstract (arXiv)
A major challenge in Reinforcement Learning (RL) is the difficulty of learning an optimal policy from sparse rewards. Prior works enhance online RL with conventional Imitation Learning (IL) via a handcrafted auxiliary objective, at the cost of restricting the RL policy to be sub-optimal when the offline data is generated by a non-expert policy. Instead, to better leverage valuable information in offline data, we develop Generalized Imitation Learning from Demonstration (GILD), which meta-learns an objective that distills knowledge from offline data and instills intrinsic motivation towards the optimal policy. Distinct from prior works that are exclusive to a specific RL algorithm, GILD is a flexible module intended for diverse vanilla off-policy RL algorithms. In addition, GILD introduces no domain-specific hyperparameter and minimal increase in computational cost. In four challenging MuJoCo tasks with sparse rewards, we show that three RL algorithms enhanced with GILD significantly outperform state-of-the-art methods.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 21 of its 28 references that are in the map. First posted on arXiv 13 Jan 2025.
Builds on
21 of 28 references.
References · 21
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
- 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
- 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
- 2018Meta-Gradient Reinforcement Learningcited by 156
- 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
- 2018On Learning Intrinsic Rewards for Policy Gradient Methodscited by 100
- 2022Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstrationcited by 54
- 2020Meta-Gradient Reinforcement Learning with an Objective Discovered Onlinecited by 52
- 2023Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learningcited by 45
- 2018Learning to Teach with Dynamic Loss Functionscited by 40
9 more
- 2021Hierarchical Few-Shot Imitation with Skill Transition Modelscited by 37
- 2018Learning a Prior over Intent via Meta-Inverse Reinforcement Learningcited by 36
- 2019Addressing the Loss-Metric Mismatch with Adaptive Loss Alignmentcited by 33
- 2019Guided Meta-Policy Searchcited by 26
- 2020Online Meta-Critic Learning for Off-Policy Actor-Critic Methodscited by 17
- 2023No-Regret Online Reinforcement Learning with Adversarial Losses and Transitionscited by 13
- 2022No-Regret Learning with Unbounded Losses: The Case of Logarithmic Poolingcited by 5
- 2022CEIP: Combining Explicit and Implicit Priors for Reinforcement Learning with Demonstrationscited by 4
- 2023L2T-DLN: Learning to Teach with Dynamic Loss Networkcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learning
- 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
- 2025Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning
- 2021Offline Meta-Reinforcement Learning with Online Self-Supervision
- 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
- 2024Hybrid Inverse Reinforcement Learning
- 2024SEABO: A Simple Search-Based Method for Offline Imitation Learning
- 2021Representation Matters: Offline Pretraining for Sequential Decision Making