# Reinforcement Learning for Sparse-Reward Object-Interaction Tasks in a First-person Simulated 3D Environment

arXiv: https://arxiv.org/abs/2010.15195 · HTML: https://mlsanity.com/paper/2010.15195/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Oct 2020.

## Built on by (most cited first, 8 of 8)

- [Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos](https://mlsanity.com/paper/2205.14065/) (2022)
- [An Investigation into Pre-Training Object-Centric Representations for Reinforcement Learning](https://mlsanity.com/paper/2302.04419/) (2023)
- [Composing Task Knowledge with Modular Successor Feature Approximators](https://mlsanity.com/paper/2301.12305/) (2023)
- [Object-Centric World Models for Causality-Aware Reinforcement Learning](https://mlsanity.com/paper/2511.14262/) (2025)
- [Learning Parameterized Task Structure for Generalization to Unseen Entities](https://mlsanity.com/paper/2203.15034/) (2022)
- [Reinforcement Learning With Sparse-Executing Actions via Sparsity Regularization](https://mlsanity.com/paper/2105.08666/) (2021)
- [Preemptive Solving of Future Problems: Multitask Preplay in Humans and Machines](https://mlsanity.com/paper/2507.05561/) (2025)
- [Feature-Attending Recurrent Modules for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2112.08369/) (2021)

## Abstract (arXiv)

First-person object-interaction tasks in high-fidelity, 3D, simulated environments such as the AI2Thor virtual home-environment pose significant sample-efficiency challenges for reinforcement learning (RL) agents learning from sparse task rewards. To alleviate these challenges, prior work has provided extensive supervision via a combination of reward-shaping, ground-truth object-information, and expert demonstrations. In this work, we show that one can learn object-interaction tasks from scratch without supervision by learning an attentive object-model as an auxiliary task during task learning with an object-centric relational RL agent. Our key insight is that learning an object-model that incorporates object-attention into forward prediction provides a dense learning signal for unsupervised representation learning of both objects and their relationships. This, in turn, enables faster policy learning for an object-centric relational RL agent. We demonstrate our agent by introducing a set of challenging object-interaction tasks in the AI2Thor environment where learning with our attentive object-model is key to strong performance. Specifically, we compare our agent and relational RL agents with alternative auxiliary tasks to a relational RL agent equipped with ground-truth object-information, and show that learning with our object-model best closes the performance gap in terms of both learning speed and maximum success rate. Additionally, we find that incorporating object-attention into an object-model's forward predictions is key to learning representations which capture object-category and object-state.