# Learning from Pixels with Expert Observations

arXiv: https://arxiv.org/abs/2306.13872 · HTML: https://mlsanity.com/paper/2306.13872/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 38 references that are in the map.
- First posted on arXiv 24 Jun 2023.

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [General E(2)-Equivariant Steerable CNNs](https://mlsanity.com/paper/1911.08251/) (2019)
- [Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning](https://mlsanity.com/paper/1910.11956/) (2019)
- [Steerable CNNs](https://mlsanity.com/paper/1612.08498/) (2016)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [Visual Reinforcement Learning with Imagined Goals](https://mlsanity.com/paper/1807.04742/) (2018)
- [Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation](https://mlsanity.com/paper/2112.05124/) (2021)
- [Frame Averaging for Invariant and Equivariant Network Design](https://mlsanity.com/paper/2110.03336/) (2021)
- [Planning with Goal-Conditioned Policies](https://mlsanity.com/paper/1911.08453/) (2019)
- [MDP Homomorphic Networks: Group Symmetries in Reinforcement Learning](https://mlsanity.com/paper/2006.16908/) (2020)
- [Learning Open Set Network with Discriminative Reciprocal Points](https://mlsanity.com/paper/2011.00178/) (2020)
- [SO(2)-Equivariant Reinforcement Learning](https://mlsanity.com/paper/2203.04439/) (2022)
- [ViNG: Learning Open-World Navigation with Visual Goals](https://mlsanity.com/paper/2012.09812/) (2020)
- [Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapes](https://mlsanity.com/paper/2110.06192/) (2021)
- [Equivariant Q Learning in Spatial Action Spaces](https://mlsanity.com/paper/2110.15443/) (2021)
- [Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning](https://mlsanity.com/paper/1912.11032/) (2019)
- [Sample Efficient Grasp Learning Using Equivariant Models](https://mlsanity.com/paper/2202.09468/) (2022)
- [Few-Shot Open-Set Recognition using Meta-Learning](https://mlsanity.com/paper/2005.13713/) (2020)
- [Spin-Weighted Spherical CNNs](https://mlsanity.com/paper/2006.10731/) (2020)
- [Deep Imitation Learning for Bimanual Robotic Manipulation](https://mlsanity.com/paper/2010.05134/) (2020)
- [Policy learning in SE(3) action spaces](https://mlsanity.com/paper/2010.02798/) (2020)
- [Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulation](https://mlsanity.com/paper/2112.00597/) (2021)
- [Manipulator-Independent Representations for Visual Imitation](https://mlsanity.com/paper/2103.09016/) (2021)
- [How to Spend Your Robot Time: Bridging Kickstarting and Offline Reinforcement Learning for Vision-based Robotic Manipulation](https://mlsanity.com/paper/2205.03353/) (2022)
- [BulletArm: An Open-Source Robotic Manipulation Benchmark and Learning Framework](https://mlsanity.com/paper/2205.14292/) (2022)
- [Hierarchical Reinforcement Learning under Mixed Observability](https://mlsanity.com/paper/2204.00898/) (2022)

## Built on by (most cited first, 3 of 3)

- [Imitate the Good and Avoid the Bad: An Incremental Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/2312.10385/) (2023)
- [SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learning](https://mlsanity.com/paper/2402.13147/) (2024)
- [Hierarchical Reinforcement Learning for Articulated Tool Manipulation with Multifingered Hand](https://mlsanity.com/paper/2507.06822/) (2025)

## Abstract (arXiv)

In reinforcement learning (RL), sparse rewards can present a significant challenge. Fortunately, expert actions can be utilized to overcome this issue. However, acquiring explicit expert actions can be costly, and expert observations are often more readily available. This paper presents a new approach that uses expert observations for learning in robot manipulation tasks with sparse rewards from pixel observations. Specifically, our technique involves using expert observations as intermediate visual goals for a goal-conditioned RL agent, enabling it to complete a task by successively reaching a series of goals. We demonstrate the efficacy of our method in five challenging block construction tasks in simulation and show that when combined with two state-of-the-art agents, our approach can significantly improve their performance while requiring 4-20 times fewer expert actions during training. Moreover, our method is also superior to a hierarchical baseline.