Learning from Pixels with Expert Observations
Not on the map. Why?
Abstract (arXiv)
In reinforcement learning (RL), sparse rewards can present a significant challenge. Fortunately, expert actions can be utilized to overcome this issue. However, acquiring explicit expert actions can be costly, and expert observations are often more readily available. This paper presents a new approach that uses expert observations for learning in robot manipulation tasks with sparse rewards from pixel observations. Specifically, our technique involves using expert observations as intermediate visual goals for a goal-conditioned RL agent, enabling it to complete a task by successively reaching a series of goals. We demonstrate the efficacy of our method in five challenging block construction tasks in simulation and show that when combined with two state-of-the-art agents, our approach can significantly improve their performance while requiring 4-20 times fewer expert actions during training. Moreover, our method is also superior to a hierarchical baseline.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 31 of its 38 references that are in the map. First posted on arXiv 24 Jun 2023.
Builds on
31 of 38 references.
References · 31
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2016Group Equivariant Convolutional Networkscited by 1,154
- 2017Hindsight Experience Replaycited by 1,096
- 2021Generalized Out-of-Distribution Detection: A Surveycited by 796
- 2019General E(2)-Equivariant Steerable CNNscited by 441
- 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
- 2016Steerable CNNscited by 298
- 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 282
- 2018Visual Reinforcement Learning with Imagined Goalscited by 248
- 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 204
- 2021Frame Averaging for Invariant and Equivariant Network Designcited by 166
19 more
- 2019Planning with Goal-Conditioned Policiescited by 144
- 2020MDP Homomorphic Networks: Group Symmetries in Reinforcement Learningcited by 133
- 2020Learning Open Set Network with Discriminative Reciprocal Pointscited by 115
- 2022SO(2)-Equivariant Reinforcement Learningcited by 79
- 2020ViNG: Learning Open-World Navigation with Visual Goalscited by 77
- 2021Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapescited by 70
- 2021Equivariant Q Learning in Spatial Action Spacescited by 61
- 2019Towards Practical Multi-Object Manipulation using Relational Reinforcement Learningcited by 54
- 2022Sample Efficient Grasp Learning Using Equivariant Modelscited by 52
- 2020Few-Shot Open-Set Recognition using Meta-Learningcited by 50
- 2020Spin-Weighted Spherical CNNscited by 50
- 2020Deep Imitation Learning for Bimanual Robotic Manipulationcited by 49
- 2020Policy learning in SE(3) action spacescited by 14
- 2021Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulationcited by 13
- 2021Manipulator-Independent Representations for Visual Imitationcited by 11
- 2022How to Spend Your Robot Time: Bridging Kickstarting and Offline Reinforcement Learning for Vision-based Robotic Manipulationcited by 9
- 2022BulletArm: An Open-Source Robotic Manipulation Benchmark and Learning Frameworkcited by 9
- 2022Hierarchical Reinforcement Learning under Mixed Observabilitycited by 4
- 2022Graph-Structured Policy Learning for Multi-Goal Manipulation Taskscited by 1
Built on by
Citing papers per year: 2023: 1; 2024: 1.
2 papers.