# MarsExplorer: Exploration of Unknown Terrains via Deep Reinforcement Learning and Procedurally Generated Environments

arXiv: https://arxiv.org/abs/2107.09996 · HTML: https://mlsanity.com/paper/2107.09996/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 36 references that are in the map.
- First posted on arXiv 21 Jul 2021.

## Builds on (most cited first, 14 of 14)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [Emergent Tool Use From Multi-Agent Autocurricula](https://mlsanity.com/paper/1909.07528/) (2019)
- [The Ingredients of Real-World Robotic Reinforcement Learning](https://mlsanity.com/paper/2004.12570/) (2020)
- [Data-efficient Deep Reinforcement Learning for Dexterous Manipulation](https://mlsanity.com/paper/1704.03073/) (2017)
- [RLlib: Abstractions for Distributed Reinforcement Learning](https://mlsanity.com/paper/1712.09381/) (2017)
- [Extending the OpenAI Gym for robotics: a toolkit for reinforcement learning using ROS and Gazebo](https://mlsanity.com/paper/1608.05742/) (2016)
- [A Multi-Resolution Frontier-Based Planner for Autonomous 3D Exploration](https://mlsanity.com/paper/2011.02182/) (2020)
- [gym-gazebo2, a toolkit for reinforcement learning using ROS 2 and Gazebo](https://mlsanity.com/paper/1903.06278/) (2019)
- [Autonomous and cooperative design of the monitor positions for a team of UAVs to maximize the quantity and quality of detected objects](https://mlsanity.com/paper/2007.01247/) (2020)

## Built on by (most cited first, 4 of 4)

- [A Survey of Zero-shot Generalisation in Deep Reinforcement Learning](https://mlsanity.com/paper/2111.09794/) (2021)
- [Space Robotics Bench: Robot Learning Beyond Earth](https://mlsanity.com/paper/2509.23328/) (2025)
- [Sim2Dust: Mastering Dynamic Waypoint Tracking on Granular Media](https://mlsanity.com/paper/2508.11503/) (2025)
- [Learning Tool-Aware Adaptive Compliant Control for Autonomous Regolith Excavation](https://mlsanity.com/paper/2509.05475/) (2025)

## Abstract (arXiv)

This paper is an initial endeavor to bridge the gap between powerful Deep Reinforcement Learning methodologies and the problem of exploration/coverage of unknown terrains. Within this scope, MarsExplorer, an openai-gym compatible environment tailored to exploration/coverage of unknown areas, is presented. MarsExplorer translates the original robotics problem into a Reinforcement Learning setup that various off-the-shelf algorithms can tackle. Any learned policy can be straightforwardly applied to a robotic platform without an elaborate simulation model of the robot's dynamics to apply a different learning/adaptation phase. One of its core features is the controllable multi-dimensional procedural generation of terrains, which is the key for producing policies with strong generalization capabilities. Four different state-of-the-art RL algorithms (A3C, PPO, Rainbow, and SAC) are trained on the MarsExplorer environment, and a proper evaluation of their results compared to the average human-level performance is reported. In the follow-up experimental analysis, the effect of the multi-dimensional difficulty setting on the learning capabilities of the best-performing algorithm (PPO) is analyzed. A milestone result is the generation of an exploration policy that follows the Hilbert curve without providing this information to the environment or rewarding directly or indirectly Hilbert-curve-like trajectories. The experimental analysis is concluded by evaluating PPO learned policy algorithm side-by-side with frontier-based exploration strategies. A study on the performance curves revealed that PPO-based policy was capable of performing adaptive-to-the-unknown-terrain sweeping without leaving expensive-to-revisit areas uncovered, underlying the capability of RL-based methodologies to tackle exploration tasks efficiently. The source code can be found at: https://github.com/dimikout3/MarsExplorer.