# TrackletMapper: Ground Surface Segmentation and Mapping from Traffic Participant Trajectories

arXiv: https://arxiv.org/abs/2209.05247 · HTML: https://mlsanity.com/paper/2209.05247/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 37 references that are in the map.
- First posted on arXiv 12 Sep 2022.

## Builds on (most cited first, 19 of 19)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Self-Supervised Visual Terrain Classification from Unsupervised Acoustic Feature Learning](https://mlsanity.com/paper/1912.03227/) (2019)
- [Find Your Own Way: Weakly-Supervised Segmentation of Path Proposals for Urban Autonomy](https://mlsanity.com/paper/1610.01238/) (2016)
- [Detecting Unexpected Obstacles for Self-Driving Cars: Fusing Deep Learning and Geometric Modeling](https://mlsanity.com/paper/1612.06573/) (2016)
- [Self-Supervised Drivable Area and Road Anomaly Segmentation using RGB-D Data for Robotic Wheelchairs](https://mlsanity.com/paper/2007.05950/) (2020)
- [Probabilistic Semantic Mapping for Urban Autonomous Driving Applications](https://mlsanity.com/paper/2006.04894/) (2020)
- [Multimodal Interaction-aware Motion Prediction for Autonomous Street Crossing](https://mlsanity.com/paper/1808.06887/) (2018)
- [DeepTemporalSeg: Temporally Consistent Semantic Segmentation of 3D LiDAR Scans](https://mlsanity.com/paper/1906.06962/) (2019)
- [Robust Monocular Localization in Sparse HD Maps Leveraging Multi-Task Uncertainty Estimation](https://mlsanity.com/paper/2110.10563/) (2021)
- [Self-Supervised Moving Vehicle Detection from Audio-Visual Cues](https://mlsanity.com/paper/2201.12771/) (2022)
- [Deep Spatiotemporal Models for Robust Proprioceptive Terrain Classification](https://mlsanity.com/paper/1804.00736/) (2018)
- [Active Bayesian Multi-class Mapping from Range and Semantic Segmentation Observation](https://mlsanity.com/paper/2101.01831/) (2021)
- [Why did the Robot Cross the Road? - Learning from Multi-Modal Sensor Data for Autonomous Road Crossing](https://mlsanity.com/paper/1709.06039/) (2017)

## Built on by (most cited first, 3 of 3)

- [Learning and Aggregating Lane Graphs for Urban Automated Driving](https://mlsanity.com/paper/2302.06175/) (2023)
- [Perception Matters: Enhancing Embodied AI with Uncertainty-Aware Semantic Segmentation](https://mlsanity.com/paper/2408.02297/) (2024)
- [AutoGraph: Predicting Lane Graphs from Traffic Observations](https://mlsanity.com/paper/2306.15410/) (2023)

## Abstract (arXiv)

Robustly classifying ground infrastructure such as roads and street crossings is an essential task for mobile robots operating alongside pedestrians. While many semantic segmentation datasets are available for autonomous vehicles, models trained on such datasets exhibit a large domain gap when deployed on robots operating in pedestrian spaces. Manually annotating images recorded from pedestrian viewpoints is both expensive and time-consuming. To overcome this challenge, we propose TrackletMapper, a framework for annotating ground surface types such as sidewalks, roads, and street crossings from object tracklets without requiring human-annotated data. To this end, we project the robot ego-trajectory and the paths of other traffic participants into the ego-view camera images, creating sparse semantic annotations for multiple types of ground surfaces from which a ground segmentation model can be trained. We further show that the model can be self-distilled for additional performance benefits by aggregating a ground surface map and projecting it into the camera images, creating a denser set of training annotations compared to the sparse tracklet annotations. We qualitatively and quantitatively attest our findings on a novel large-scale dataset for mobile robots operating in pedestrian areas. Code and dataset will be made available at http://trackletmapper.cs.uni-freiburg.de.