# MotionSC: Data Set and Network for Real-Time Semantic Mapping in Dynamic Environments

arXiv: https://arxiv.org/abs/2203.07060 · HTML: https://mlsanity.com/paper/2203.07060/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in World modeling from LiDAR generation, one of 11 directions in World models.
- Builds on 46 of its 62 references that are in the map.
- First posted on arXiv 14 Mar 2022.

## Placed in

- World models / [World modeling from LiDAR generation](https://mlsanity.com/wm/world-modeling-from-lidar-generation/)

## Builds on (most cited first, 30 of 46)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [Deep Learning for 3D Point Clouds: A Survey](https://mlsanity.com/paper/1912.12033/) (2019)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1611.08974/) (2016)
- [SqueezeSeg: Convolutional Neural Nets with Recurrent CRF for Real-Time Road-Object Segmentation from 3D LiDAR Point Cloud](https://mlsanity.com/paper/1710.07368/) (2017)
- [Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completion](https://mlsanity.com/paper/2012.03762/) (2020)
- [MP3: A Unified Model to Map, Perceive, Predict and Plan](https://mlsanity.com/paper/2101.06806/) (2021)
- [LMSCNet: Lightweight Multiscale 3D Semantic Completion](https://mlsanity.com/paper/2008.10559/) (2020)
- [Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Art](https://mlsanity.com/paper/1704.05519/) (2017)
- [SemanticPOSS: A Point Cloud Dataset with Large Quantity of Dynamic Instances](https://mlsanity.com/paper/2002.09147/) (2020)
- [S3CNet: A Sparse Semantic Scene Completion Network for LiDAR Point Clouds](https://mlsanity.com/paper/2012.09242/) (2020)
- [MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird's Eye View Maps](https://mlsanity.com/paper/2003.06754/) (2020)
- [Virtual-to-real Deep Reinforcement Learning: Continuous Control of Mobile Robots for Mapless Navigation](https://mlsanity.com/paper/1703.00420/) (2017)
- [Deep Projective 3D Semantic Segmentation](https://mlsanity.com/paper/1705.03428/) (2017)
- [Semantic Scene Completion using Local Deep Implicit Functions on LiDAR Data](https://mlsanity.com/paper/2011.09141/) (2020)
- [3D Semantic Scene Completion: a Survey](https://mlsanity.com/paper/2103.07466/) (2021)
- [Learning Navigation Behaviors End-to-End with AutoRL](https://mlsanity.com/paper/1809.10124/) (2018)

## Built on by (most cited first, 30 of 40)

- [SCPNet: Semantic Scene Completion on Point Cloud](https://mlsanity.com/paper/2303.06884/) (2023)
- [3D and 4D World Modeling: A Survey](https://mlsanity.com/paper/2509.07996/) (2025)
- [A Survey of World Models for Autonomous Driving](https://mlsanity.com/paper/2501.11260/) (2025)
- [SemCity: Semantic Scene Generation with Triplane Diffusion](https://mlsanity.com/paper/2403.07773/) (2024)
- [Pyramid Diffusion for Fine 3D Large Scene Generation](https://mlsanity.com/paper/2311.12085/) (2023)
- [Not All Voxels Are Equal: Hardness-Aware Semantic Scene Completion with Self-Distillation](https://mlsanity.com/paper/2404.11958/) (2024)
- [DynamicCity: Large-Scale 4D Occupancy Generation from Dynamic Scenes](https://mlsanity.com/paper/2410.18084/) (2024)
- [Diffusion Probabilistic Models for Scene-Scale 3D Categorical Data](https://mlsanity.com/paper/2301.00527/) (2023)
- [OccMamba: Semantic Occupancy Prediction with State Space Models](https://mlsanity.com/paper/2408.09859/) (2024)
- [CRKD: Enhanced Camera-Radar Object Detection with Cross-modality Knowledge Distillation](https://mlsanity.com/paper/2403.19104/) (2024)
- [UniOcc: A Unified Benchmark for Occupancy Forecasting and Prediction in Autonomous Driving](https://mlsanity.com/paper/2503.24381/) (2025)
- [Grid-Centric Traffic Scenario Perception for Autonomous Driving: A Comprehensive Review](https://mlsanity.com/paper/2303.01212/) (2023)
- [3D Scene Generation: A Survey](https://mlsanity.com/paper/2505.05474/) (2025)
- [NuiScene: Exploring Efficient Generation of Unbounded Outdoor Scenes](https://mlsanity.com/paper/2503.16375/) (2025)
- [GaussianFormer3D: Multi-Modal Gaussian-based Semantic Occupancy Prediction with 3D Deformable Attention](https://mlsanity.com/paper/2505.10685/) (2025)
- [ConvBKI: Real-Time Probabilistic Semantic Mapping Network with Quantifiable Uncertainty](https://mlsanity.com/paper/2310.16020/) (2023)
- [LiRaFusion: Deep Adaptive LiDAR-Radar Fusion for 3D Object Detection](https://mlsanity.com/paper/2402.11735/) (2024)
- [La La LiDAR: Large-Scale Layout Generation from LiDAR Data](https://mlsanity.com/paper/2508.03691/) (2025)
- [Label-efficient Semantic Scene Completion with Scribble Annotations](https://mlsanity.com/paper/2405.15170/) (2024)
- [PanoSSC: Exploring Monocular Panoptic 3D Scene Reconstruction for Autonomous Driving](https://mlsanity.com/paper/2406.07037/) (2024)
- [MemFusionMap: Working Memory Fusion for Online Vectorized HD Map Construction](https://mlsanity.com/paper/2409.18737/) (2024)
- [Controllable 3D Outdoor Scene Generation via Scene Graphs](https://mlsanity.com/paper/2503.07152/) (2025)
- [Convolutional Bayesian Kernel Inference for 3D Semantic Mapping](https://mlsanity.com/paper/2209.10663/) (2022)
- [BeautyMap: Binary-Encoded Adaptable Ground Matrix for Dynamic Points Removal in Global Maps](https://mlsanity.com/paper/2405.07283/) (2024)
- [ReliOcc: Towards Reliable Semantic Occupancy Prediction via Uncertainty Learning](https://mlsanity.com/paper/2409.18026/) (2024)
- [UFO: Uncertainty-aware LiDAR-image Fusion for Off-road Semantic Terrain Map Estimation](https://mlsanity.com/paper/2403.02642/) (2024)
- [Towards Generating Realistic 3D Semantic Training Data for Autonomous Driving](https://mlsanity.com/paper/2503.21449/) (2025)
- [Extend3D: Town-Scale 3D Generation](https://mlsanity.com/paper/2603.29387/) (2026)
- [Deep Learning for 3D Point Cloud Enhancement: A Survey](https://mlsanity.com/paper/2411.00857/) (2024)
- [Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AI](https://mlsanity.com/paper/2510.04978/) (2025)

## Abstract (arXiv)

This work addresses a gap in semantic scene completion (SSC) data by creating a novel outdoor data set with accurate and complete dynamic scenes. Our data set is formed from randomly sampled views of the world at each time step, which supervises generalizability to complete scenes without occlusions or traces. We create SSC baselines from state-of-the-art open source networks and construct a benchmark real-time dense local semantic mapping algorithm, MotionSC, by leveraging recent 3D deep learning architectures to enhance SSC with temporal information. Our network shows that the proposed data set can quantify and supervise accurate scene completion in the presence of dynamic objects, which can lead to the development of improved dynamic mapping algorithms. All software is available at https://github.com/UMich-CURLY/3DMapping.