# Out of the Room: Generalizing Event-Based Dynamic Motion Segmentation for Complex Scenes

arXiv: https://arxiv.org/abs/2403.04562 · HTML: https://mlsanity.com/paper/2403.04562/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 58 references that are in the map.
- First posted on arXiv 7 Mar 2024.

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Age](https://mlsanity.com/paper/1606.05830/) (2016)
- [Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion](https://mlsanity.com/paper/1812.08156/) (2018)
- [DSEC: A Stereo Event Camera Dataset for Driving Scenarios](https://mlsanity.com/paper/2103.06011/) (2021)
- [EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras](https://mlsanity.com/paper/1802.06898/) (2018)
- [End-to-End Learning of Representations for Asynchronous Event-Based Data](https://mlsanity.com/paper/1904.08245/) (2019)
- [Learning to Detect Objects with a 1 Megapixel Event Camera](https://mlsanity.com/paper/2009.13436/) (2020)
- [A Unifying Contrast Maximization Framework for Event Cameras, with Applications to Motion, Depth, and Optical Flow Estimation](https://mlsanity.com/paper/1804.01306/) (2018)
- [E-RAFT: Dense Optical Flow from Event Cameras](https://mlsanity.com/paper/2108.10552/) (2021)
- [TimeLens: Event-based Video Frame Interpolation](https://mlsanity.com/paper/2106.07286/) (2021)
- [Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance](https://mlsanity.com/paper/2007.06936/) (2020)
- [Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphs](https://mlsanity.com/paper/2101.06894/) (2021)
- [Event-based Moving Object Detection and Tracking](https://mlsanity.com/paper/1803.04523/) (2018)
- [AEGNN: Asynchronous Event-based Graph Neural Networks](https://mlsanity.com/paper/2203.17149/) (2022)
- [Self-Supervised Learning of Event-Based Optical Flow with Spiking Neural Networks](https://mlsanity.com/paper/2106.01862/) (2021)
- [Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks](https://mlsanity.com/paper/2003.06696/) (2020)
- [Secrets of Event-Based Optical Flow](https://mlsanity.com/paper/2207.10022/) (2022)
- [Learning Monocular Dense Depth from Events](https://mlsanity.com/paper/2010.08350/) (2020)
- [Unsupervised Monocular Depth Learning in Dynamic Scenes](https://mlsanity.com/paper/2010.16404/) (2020)
- [ESS: Learning Event-based Semantic Segmentation from Still Images](https://mlsanity.com/paper/2203.10016/) (2022)
- [Event-Based Motion Segmentation by Motion Compensation](https://mlsanity.com/paper/1904.01293/) (2019)
- [It's Moving! A Probabilistic Model for Causal Motion Segmentation in Moving Camera Videos](https://mlsanity.com/paper/1604.00136/) (2016)
- [Unsupervised Scale-consistent Depth Learning from Video](https://mlsanity.com/paper/2105.11610/) (2021)
- [Segmenting Moving Objects via an Object-Centric Layered Representation](https://mlsanity.com/paper/2207.02206/) (2022)

## Built on by (most cited first, 7 of 7)

- [Instance-Level Moving Object Segmentation from a Single Image with Events](https://mlsanity.com/paper/2502.12975/) (2025)
- [On-Device Self-Supervised Learning of Low-Latency Monocular Depth from Only Events](https://mlsanity.com/paper/2412.06359/) (2024)
- [Hardware, Algorithms, and Applications of the Neuromorphic Vision Sensor: a Review](https://mlsanity.com/paper/2504.08588/) (2025)
- [Motion Segmentation for Neuromorphic Aerial Surveillance](https://mlsanity.com/paper/2405.15209/) (2024)
- [Iterative Event-based Motion Segmentation by Variational Contrast Maximization](https://mlsanity.com/paper/2504.18447/) (2025)
- [Event-based Optical Marker Systems: A survey](https://mlsanity.com/paper/2504.20736/) (2025)
- [EV-LayerSegNet: Self-supervised Motion Segmentation using Event Cameras](https://mlsanity.com/paper/2506.06596/) (2025)

## Abstract (arXiv)

Rapid and reliable identification of dynamic scene parts, also known as motion segmentation, is a key challenge for mobile sensors. Contemporary RGB camera-based methods rely on modeling camera and scene properties however, are often under-constrained and fall short in unknown categories. Event cameras have the potential to overcome these limitations, but corresponding methods have only been demonstrated in smaller-scale indoor environments with simplified dynamic objects. This work presents an event-based method for class-agnostic motion segmentation that can successfully be deployed across complex large-scale outdoor environments too. To this end, we introduce a novel divide-and-conquer pipeline that combines: (a) ego-motion compensated events, computed via a scene understanding module that predicts monocular depth and camera pose as auxiliary tasks, and (b) optical flow from a dedicated optical flow module. These intermediate representations are then fed into a segmentation module that predicts motion segmentation masks. A novel transformer-based temporal attention module in the segmentation module builds correlations across adjacent 'frames' to get temporally consistent segmentation masks. Our method sets the new state-of-the-art on the classic EV-IMO benchmark (indoors), where we achieve improvements of 2.19 moving object IoU (2.22 mIoU) and 4.52 point IoU respectively, as well as on a newly-generated motion segmentation and tracking benchmark (outdoors) based on the DSEC event dataset, termed DSEC-MOTS, where we show improvement of 12.91 moving object IoU.