# An Open-Source Multi-Goal Reinforcement Learning Environment for Robotic Manipulation with Pybullet

arXiv: https://arxiv.org/abs/2105.05985 · HTML: https://mlsanity.com/paper/2105.05985/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 18 references that are in the map.
- First posted on arXiv 12 May 2021.

## Builds on (most cited first, 12 of 12)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](https://mlsanity.com/paper/2003.04960/) (2020)
- [Challenges of Real-World Reinforcement Learning](https://mlsanity.com/paper/1904.12901/) (2019)
- [Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research](https://mlsanity.com/paper/1802.09464/) (2018)
- [iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenes](https://mlsanity.com/paper/2012.02924/) (2020)
- [On The Effect of Auxiliary Tasks on Representation Dynamics](https://mlsanity.com/paper/2102.13089/) (2021)
- [Optimistic Exploration even with a Pessimistic Initialisation](https://mlsanity.com/paper/2002.12174/) (2020)
- [Curriculum Learning with Hindsight Experience Replay for Sequential Object Manipulation Tasks](https://mlsanity.com/paper/2008.09377/) (2020)

## Built on by (most cited first, 7 of 7)

- [RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learning](https://mlsanity.com/paper/2504.18904/) (2025)
- [Relay Hindsight Experience Replay: Self-Guided Continual Reinforcement Learning for Sequential Object Manipulation Tasks with Sparse Rewards](https://mlsanity.com/paper/2208.00843/) (2022)
- [Train a Real-world Local Path Planner in One Hour via Partially Decoupled Reinforcement Learning and Vectorized Diversity](https://mlsanity.com/paper/2305.04180/) (2023)
- [Abstract Demonstrations and Adaptive Exploration for Efficient and Stable Multi-step Sparse Reward Reinforcement Learning](https://mlsanity.com/paper/2207.09243/) (2022)
- [Sample-efficient Low-level Motion Planning for Robotic Manipulation Tasks via Zero-shot Transfer Learning](https://mlsanity.com/paper/2606.06041/) (2026)
- [Hybrid Robot Learning for Automatic Robot Motion Planning in Manufacturing](https://mlsanity.com/paper/2502.19340/) (2025)
- [Enhancing RL Generalizability in Robotics through SHAP Analysis of Algorithms and Hyperparameters](https://mlsanity.com/paper/2605.02867/) (2026)

## Abstract (arXiv)

This work re-implements the OpenAI Gym multi-goal robotic manipulation environment, originally based on the commercial Mujoco engine, onto the open-source Pybullet engine. By comparing the performances of the Hindsight Experience Replay-aided Deep Deterministic Policy Gradient agent on both environments, we demonstrate our successful re-implementation of the original environment. Besides, we provide users with new APIs to access a joint control mode, image observations and goals with customisable camera and a built-in on-hand camera. We further design a set of multi-step, multi-goal, long-horizon and sparse reward robotic manipulation tasks, aiming to inspire new goal-conditioned reinforcement learning algorithms for such challenges. We use a simple, human-prior-based curriculum learning method to benchmark the multi-step manipulation tasks. Discussions about future research opportunities regarding this kind of tasks are also provided.