# Reconstructing occluded Elevation Information in Terrain Maps with Self-supervised Learning

arXiv: https://arxiv.org/abs/2109.07150 · HTML: https://mlsanity.com/paper/2109.07150/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 40 references that are in the map.
- First posted on arXiv 15 Sep 2021.

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [A Neural Algorithm of Artistic Style](https://mlsanity.com/paper/1508.06576/) (2015)
- [Self-Supervised Scene De-occlusion](https://mlsanity.com/paper/2004.02788/) (2020)
- [Image inpainting: A review](https://mlsanity.com/paper/1909.06399/) (2019)
- [SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scans](https://mlsanity.com/paper/1912.00036/) (2019)
- [Patch-Based Image Inpainting with Generative Adversarial Networks](https://mlsanity.com/paper/1803.07422/) (2018)
- [Contextual-based Image Inpainting: Infer, Match, and Translate](https://mlsanity.com/paper/1711.08590/) (2017)
- [Learning for Active 3D Mapping](https://mlsanity.com/paper/1708.02074/) (2017)
- [A GNC Architecture for Planetary Rovers with Autonomous Navigation Capabilities](https://mlsanity.com/paper/1911.09975/) (2019)

## Built on by (most cited first, 4 of 4)

- [Neural Scene Representation for Locomotion on Structured Terrain](https://mlsanity.com/paper/2206.08077/) (2022)
- [Pixel to Elevation: Learning to Predict Elevation Maps at Long Range using Images for Autonomous Offroad Navigation](https://mlsanity.com/paper/2401.17484/) (2024)
- [Input-to-State Stable Coupled Oscillator Networks for Closed-form Model-based Control in Latent Space](https://mlsanity.com/paper/2409.08439/) (2024)
- [Motion planning for off-road autonomous driving based on human-like cognition and weight adaptation](https://mlsanity.com/paper/2404.17820/) (2024)

## Abstract (arXiv)

Accurate and complete terrain maps enhance the awareness of autonomous robots and enable safe and optimal path planning. Rocks and topography often create occlusions and lead to missing elevation information in the Digital Elevation Map (DEM). Currently, these occluded areas are either fully avoided during motion planning or the missing values in the elevation map are filled-in using traditional interpolation, diffusion or patch-matching techniques. These methods cannot leverage the high-level terrain characteristics and the geometric constraints of line of sight we humans use intuitively to predict occluded areas. We introduce a self-supervised learning approach capable of training on real-world data without a need for ground-truth information to reconstruct the occluded areas in the DEMs. We accomplish this by adding artificial occlusion to the incomplete elevation maps constructed on a real robot by performing ray casting. We first evaluate a supervised learning approach on synthetic data for which we have the full ground-truth available and subsequently move to several real-world datasets. These real-world datasets were recorded during exploration of both structured and unstructured terrain with a legged robot, and additionally in a planetary scenario on Lunar analogue terrain. We state a significant improvement compared to the baseline methods both on synthetic terrain and for the real-world datasets. Our neural network is able to run in real-time on both CPU and GPU with suitable sampling rates for autonomous ground robots. We motivate the applicability of reconstructing occlusion in elevation maps with preliminary motion planning experiments.