# Non-parametric Memory for Spatio-Temporal Segmentation of Construction Zones for Self-Driving

arXiv: https://arxiv.org/abs/2101.06865 · HTML: https://mlsanity.com/paper/2101.06865/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 45 references that are in the map.
- First posted on arXiv 18 Jan 2021.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs](https://mlsanity.com/paper/1412.7062/) (2014)
- [Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture](https://mlsanity.com/paper/1411.4734/) (2014)
- [OctNet: Learning Deep 3D Representations at High Resolutions](https://mlsanity.com/paper/1611.05009/) (2016)
- [Joint 3D Proposal Generation and Object Detection from View Aggregation](https://mlsanity.com/paper/1712.02294/) (2017)
- [Deep Continuous Fusion for Multi-Sensor 3D Object Detection](https://mlsanity.com/paper/2012.10992/) (2020)
- [Conditional Random Fields as Recurrent Neural Networks](https://mlsanity.com/paper/1502.03240/) (2015)
- [Tangent Convolutions for Dense Prediction in 3D](https://mlsanity.com/paper/1807.02443/) (2018)
- [Deep Parametric Continuous Convolutional Neural Networks](https://mlsanity.com/paper/2101.06742/) (2021)
- [SPLATNet: Sparse Lattice Networks for Point Cloud Processing](https://mlsanity.com/paper/1802.08275/) (2018)
- [SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks](https://mlsanity.com/paper/1609.05130/) (2016)
- [Vehicle Detection from 3D Lidar Using Fully Convolutional Network](https://mlsanity.com/paper/1608.07916/) (2016)
- [Recurrent Slice Networks for 3D Segmentation of Point Clouds](https://mlsanity.com/paper/1802.04402/) (2018)
- [Learning Video Object Segmentation with Visual Memory](https://mlsanity.com/paper/1704.05737/) (2017)
- [Semantic Visual Localization](https://mlsanity.com/paper/1712.05773/) (2017)
- [SBNet: Sparse Blocks Network for Fast Inference](https://mlsanity.com/paper/1801.02108/) (2018)
- [Deep Multi-Sensor Lane Detection](https://mlsanity.com/paper/1905.01555/) (2019)
- [Higher Order Conditional Random Fields in Deep Neural Networks](https://mlsanity.com/paper/1511.08119/) (2015)
- [Recurrent Instance Segmentation](https://mlsanity.com/paper/1511.08250/) (2015)
- [Parallel Multi-Dimensional LSTM, With Application to Fast Biomedical Volumetric Image Segmentation](https://mlsanity.com/paper/1506.07452/) (2015)

## Built on by (most cited first, 1 of 1)

- [Improving Autonomous Vehicle Mapping and Navigation in Work Zones Using Crowdsourcing Vehicle Trajectories](https://mlsanity.com/paper/2301.09194/) (2023)

## Abstract (arXiv)

In this paper, we introduce a non-parametric memory representation for spatio-temporal segmentation that captures the local space and time around an autonomous vehicle (AV). Our representation has three important properties: (i) it remembers what it has seen in the past, (ii) it reinforces and (iii) forgets its past beliefs based on new evidence. Reinforcing is important as the first time we see an element we might be uncertain, e.g, if the element is heavily occluded or at range. Forgetting is desirable, as otherwise false positives will make the self driving vehicle behave erratically. Our process is informed by 3D reasoning, as occlusion is key to distinguishing between the desire to forget and to remember. We show how our method can be used as an online component to complement static world representations such as HD maps by detecting and remembering changes that should be superimposed on top of this static view due to such events.