# Monocular Instance Motion Segmentation for Autonomous Driving: KITTI InstanceMotSeg Dataset and Multi-task Baseline

arXiv: https://arxiv.org/abs/2008.07008 · HTML: https://mlsanity.com/paper/2008.07008/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Aug 2020.

## Built on by (most cited first, 21 of 21)

- [A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlook](https://mlsanity.com/paper/2401.01454/) (2024)
- [Segment Any Motion in Videos](https://mlsanity.com/paper/2503.22268/) (2025)
- [Surround-view Fisheye Camera Perception for Automated Driving: Overview, Survey and Challenges](https://mlsanity.com/paper/2205.13281/) (2022)
- [Near-field Perception for Low-Speed Vehicle Automation using Surround-view Fisheye Cameras](https://mlsanity.com/paper/2103.17001/) (2021)
- [Optical Flow for Autonomous Driving: Applications, Challenges and Improvements](https://mlsanity.com/paper/2301.04422/) (2023)
- [Motion Segmentation from a Moving Monocular Camera](https://mlsanity.com/paper/2309.13772/) (2023)
- [BEV-MODNet: Monocular Camera based Bird's Eye View Moving Object Detection for Autonomous Driving](https://mlsanity.com/paper/2107.04937/) (2021)
- [Zero-Shot Monocular Motion Segmentation in the Wild by Combining Deep Learning with Geometric Motion Model Fusion](https://mlsanity.com/paper/2405.01723/) (2024)
- [Woodscape Fisheye Object Detection for Autonomous Driving -- CVPR 2022 OmniCV Workshop Challenge](https://mlsanity.com/paper/2206.12912/) (2022)
- [RALACs: Action Recognition in Autonomous Vehicles using Interaction Encoding and Optical Flow](https://mlsanity.com/paper/2209.14408/) (2022)
- [DaF-BEVSeg: Distortion-aware Fisheye Camera based Bird's Eye View Segmentation with Occlusion Reasoning](https://mlsanity.com/paper/2404.06352/) (2024)
- [FisheyeDetNet: 360deg Surround view Fisheye Camera based Object Detection System for Autonomous Driving](https://mlsanity.com/paper/2404.13443/) (2024)
- [Efficient Microscopic Image Instance Segmentation for Food Crystal Quality Control](https://mlsanity.com/paper/2409.18291/) (2024)
- [Hierarchical Visual Prompt Learning for Continual Video Instance Segmentation](https://mlsanity.com/paper/2508.08612/) (2025)
- [Weakly Supervised Instance Segmentation using Motion Information via Optical Flow](https://mlsanity.com/paper/2202.13006/) (2022)
- [CORP: A Multi-Modal Dataset for Campus-Oriented Roadside Perception Tasks](https://mlsanity.com/paper/2404.03191/) (2024)
- [WoodScape Motion Segmentation for Autonomous Driving -- CVPR 2023 OmniCV Workshop Challenge](https://mlsanity.com/paper/2401.00910/) (2023)
- [Neural Rendering based Urban Scene Reconstruction for Autonomous Driving](https://mlsanity.com/paper/2402.06826/) (2024)
- [Impact of Video Compression Artifacts on Fisheye Camera Visual Perception Tasks](https://mlsanity.com/paper/2403.16338/) (2024)
- [Dense Monocular Motion Segmentation Using Optical Flow and Pseudo Depth Map: A Zero-Shot Approach](https://mlsanity.com/paper/2406.18837/) (2024)
- [MotionSync: Non-Causal Refinement of Causal Tracker for Label-Efficient 3D Perception](https://mlsanity.com/paper/2608.29567/) (2026)

## Abstract (arXiv)

Moving object segmentation is a crucial task for autonomous vehicles as it can be used to segment objects in a class agnostic manner based on their motion cues. It enables the detection of unseen objects during training (e.g., moose or a construction truck) based on their motion and independent of their appearance. Although pixel-wise motion segmentation has been studied in autonomous driving literature, it has been rarely addressed at the instance level, which would help separate connected segments of moving objects leading to better trajectory planning. As the main issue is the lack of large public datasets, we create a new InstanceMotSeg dataset comprising of 12.9K samples improving upon our KITTIMoSeg dataset. In addition to providing instance level annotations, we have added 4 additional classes which is crucial for studying class agnostic motion segmentation. We adapt YOLACT and implement a motion-based class agnostic instance segmentation model which would act as a baseline for the dataset. We also extend it to an efficient multi-task model which additionally provides semantic instance segmentation sharing the encoder. The model then learns separate prototype coefficients within the class agnostic and semantic heads providing two independent paths of object detection for redundant safety. To obtain real-time performance, we study different efficient encoders and obtain 39 fps on a Titan Xp GPU using MobileNetV2 with an improvement of 10% mAP relative to the baseline. Our model improves the previous state of the art motion segmentation method by 3.3%. The dataset and qualitative results video are shared in our website at https://sites.google.com/view/instancemotseg/.