# SLPC: a VRNN-based approach for stochastic lidar prediction and completion in autonomous driving

arXiv: https://arxiv.org/abs/2102.09883 · HTML: https://mlsanity.com/paper/2102.09883/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 33 references that are in the map.
- First posted on arXiv 19 Feb 2021.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [A Recurrent Latent Variable Model for Sequential Data](https://mlsanity.com/paper/1506.02216/) (2015)
- [Sparsity Invariant CNNs](https://mlsanity.com/paper/1708.06500/) (2017)
- [Unsupervised Learning for Physical Interaction through Video Prediction](https://mlsanity.com/paper/1605.07157/) (2016)
- [Stochastic Video Generation with a Learned Prior](https://mlsanity.com/paper/1802.07687/) (2018)
- [Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning](https://mlsanity.com/paper/1605.08104/) (2016)
- [Stochastic Adversarial Video Prediction](https://mlsanity.com/paper/1804.01523/) (2018)
- [Action-Conditional Video Prediction using Deep Networks in Atari Games](https://mlsanity.com/paper/1507.08750/) (2015)
- [Self-supervised Sparse-to-Dense: Self-supervised Depth Completion from LiDAR and Monocular Camera](https://mlsanity.com/paper/1807.00275/) (2018)
- [Video Pixel Networks](https://mlsanity.com/paper/1610.00527/) (2016)
- [Improved Conditional VRNNs for Video Prediction](https://mlsanity.com/paper/1904.12165/) (2019)
- [Learning to See by Moving](https://mlsanity.com/paper/1505.01596/) (2015)
- [Don't Forget The Past: Recurrent Depth Estimation from Monocular Video](https://mlsanity.com/paper/2001.02613/) (2020)
- [SDCNet: Video Prediction Using Spatially-Displaced Convolution](https://mlsanity.com/paper/1811.00684/) (2018)
- [Long-Term Occupancy Grid Prediction Using Recurrent Neural Networks](https://mlsanity.com/paper/1809.03782/) (2018)
- [Dynamic Environment Prediction in Urban Scenes using Recurrent Representation Learning](https://mlsanity.com/paper/1904.12374/) (2019)
- [An Adversarial Super-Resolution Remedy for Radar Design Trade-offs](https://mlsanity.com/paper/1903.01392/) (2019)
- [Person Identification and Body Mass Index: A Deep Learning-Based Study on Micro-Dopplers](https://mlsanity.com/paper/1811.07173/) (2018)
- [Inverting the Pose Forecasting Pipeline with SPF2: Sequential Pointcloud Forecasting for Sequential Pose Forecasting](https://mlsanity.com/paper/2003.08376/) (2020)
- [Towards Adversarial Denoising of Radar Micro-Doppler Signatures](https://mlsanity.com/paper/1811.04678/) (2018)
- [2D LiDAR Map Prediction via Estimating Motion Flow with GRU](https://mlsanity.com/paper/1902.06919/) (2019)
- [Stairs Detection for Enhancing Wheelchair Capabilities Based on Radar Sensors](https://mlsanity.com/paper/1711.09206/) (2017)

## Built on by (most cited first, 3 of 3)

- [HALS: A Height-Aware Lidar Super-Resolution Framework for Autonomous Driving](https://mlsanity.com/paper/2202.03901/) (2022)
- [Enhancing Vehicle Environmental Awareness via Federated Learning and Automatic Labeling](https://mlsanity.com/paper/2408.12769/) (2024)
- [Temporal Lidar Depth Completion](https://mlsanity.com/paper/2406.11315/) (2024)

## Abstract (arXiv)

Predicting future 3D LiDAR pointclouds is a challenging task that is useful in many applications in autonomous driving such as trajectory prediction, pose forecasting and decision making. In this work, we propose a new LiDAR prediction framework that is based on generative models namely Variational Recurrent Neural Networks (VRNNs), titled Stochastic LiDAR Prediction and Completion (SLPC). Our algorithm is able to address the limitations of previous video prediction frameworks when dealing with sparse data by spatially inpainting the depth maps in the upcoming frames. Our contributions can thus be summarized as follows: we introduce the new task of predicting and completing depth maps from spatially sparse data, we present a sparse version of VRNNs and an effective self-supervised training method that does not require any labels. Experimental results illustrate the effectiveness of our framework in comparison to the state of the art methods in video prediction.