Efficient Imitation Without Demonstrations via Value-Penalized Auxiliary Control from Examples
Not on the map. Why?
Abstract (arXiv)
Common approaches to providing feedback in reinforcement learning are the use of hand-crafted rewards or full-trajectory expert demonstrations. Alternatively, one can use examples of completed tasks, but such an approach can be extremely sample inefficient. We introduce value-penalized auxiliary control from examples (VPACE), an algorithm that significantly improves exploration in example-based control by adding examples of simple auxiliary tasks and an above-success-level value penalty. Across both simulated and real robotic environments, we show that our approach substantially improves learning efficiency for challenging tasks, while maintaining bounded value estimates. Preliminary results also suggest that VPACE may learn more efficiently than the more common approaches of using full trajectories or true sparse rewards. Project site: https://papers.starslab.ca/vpace/.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 32 of its 48 references that are in the map. First posted on arXiv 3 Jul 2024.
Builds on
32 of 48 references.
References · 32
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
- 2016Generative Adversarial Imitation Learningcited by 1,576
- 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
- 2017Hindsight Experience Replaycited by 1,096
- 2018Exploration by Random Network Distillationcited by 966
- 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
- 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
- 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
- 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
- 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
- 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
20 more
- 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
- 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 203
- 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
- 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 164
- 2021Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisationcited by 120
- 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
- 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
- 2017Time Limits in Reinforcement Learningcited by 72
- 2018Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definitioncited by 66
- 2019Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?cited by 66
- 2017Data-efficient Deep Reinforcement Learning for Dexterous Manipulationcited by 61
- 2022Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexitycited by 59
- 2021Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classificationcited by 38
- 2021Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Taskscited by 12
- 2024Revisiting Sparse Rewards for Goal-Reaching Reinforcement Learningcited by 12
- 2022Learning from Guided Play: Improving Exploration for Adversarial Imitation Learning with Simple Auxiliary Taskscited by 10
- 2024Boosting Soft Q-Learning by Boundingcited by 4
- 2021Learning from Guided Play: A Scheduled Hierarchical Approach for Improving Exploration in Adversarial Imitation Learningcited by 3
- 2023Contrastive Example-Based Controlcited by 3
- 2024Is Exploration All You Need? Effective Exploration Characteristics for Transfer in Reinforcement Learningcited by 3
Built on by
Citing papers per year: 2025: 1.
1 paper.