# Neuromorphic Vision-based Motion Segmentation with Graph Transformer Neural Network

arXiv: https://arxiv.org/abs/2404.10940 · HTML: https://mlsanity.com/paper/2404.10940/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 63 references that are in the map.
- First posted on arXiv 16 Apr 2024.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [PCT: Point cloud transformer](https://mlsanity.com/paper/2012.09688/) (2020)
- [Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling](https://mlsanity.com/paper/2111.14819/) (2021)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [Exploring Self-attention for Image Recognition](https://mlsanity.com/paper/2004.13621/) (2020)
- [Recurrent Vision Transformers for Object Detection with Event Cameras](https://mlsanity.com/paper/2212.05598/) (2022)
- [Ultimate SLAM? Combining Events, Images, and IMU for Robust Visual SLAM in HDR and High Speed Scenarios](https://mlsanity.com/paper/1709.06310/) (2017)
- [A Survey on Deep Learning Technique for Video Segmentation](https://mlsanity.com/paper/2107.01153/) (2021)
- [Event-based Moving Object Detection and Tracking](https://mlsanity.com/paper/1803.04523/) (2018)
- [Video Object Segmentation with Episodic Graph Memory Networks](https://mlsanity.com/paper/2007.07020/) (2020)
- [Event-aided Direct Sparse Odometry](https://mlsanity.com/paper/2204.07640/) (2022)
- [Zero-Shot Video Object Segmentation via Attentive Graph Neural Networks](https://mlsanity.com/paper/2001.06807/) (2020)
- [Focus Is All You Need: Loss Functions For Event-based Vision](https://mlsanity.com/paper/1904.07235/) (2019)
- [Event-Based Motion Segmentation by Motion Compensation](https://mlsanity.com/paper/1904.01293/) (2019)
- [Event-based Motion Segmentation with Spatio-Temporal Graph Cuts](https://mlsanity.com/paper/2012.08730/) (2020)
- [Event Transformer. A sparse-aware solution for efficient event data processing](https://mlsanity.com/paper/2204.03355/) (2022)
- [EV-IMO: Motion Segmentation Dataset and Learning Pipeline for Event Cameras](https://mlsanity.com/paper/1903.07520/) (2019)
- [DilateFormer: Multi-Scale Dilated Transformer for Visual Recognition](https://mlsanity.com/paper/2302.01791/) (2023)
- [EVIMO2: An Event Camera Dataset for Motion Segmentation, Optical Flow, Structure from Motion, and Visual Inertial Odometry in Indoor Scenes with Monocular or Stereo Algorithms](https://mlsanity.com/paper/2205.03467/) (2022)
- [SpikeMS: Deep Spiking Neural Network for Motion Segmentation](https://mlsanity.com/paper/2105.06562/) (2021)
- [Dual Transformer for Point Cloud Analysis](https://mlsanity.com/paper/2104.13044/) (2021)
- [Real-time clustering and multi-target tracking using event-based sensors](https://mlsanity.com/paper/1807.02851/) (2018)
- [Neuromorphic Camera Denoising using Graph Neural Network-driven Transformers](https://mlsanity.com/paper/2112.09685/) (2021)
- [A Neuromorphic Vision-Based Measurement for Robust Relative Localization in Future Space Exploration Missions](https://mlsanity.com/paper/2206.11541/) (2022)
- [A Survey on Evolutionary Computation for Computer Vision and Image Analysis: Past, Present, and Future Trends](https://mlsanity.com/paper/2209.06399/) (2022)
- [SalientDSO: Bringing Attention to Direct Sparse Odometry](https://mlsanity.com/paper/1803.00127/) (2018)
- [Neuromorphic Vision Based Control for the Precise Positioning of Robotic Drilling Systems](https://mlsanity.com/paper/2201.01626/) (2022)
- [Asynchronous Events-based Panoptic Segmentation using Graph Mixer Neural Network](https://mlsanity.com/paper/2305.03640/) (2023)

## Built on by (most cited first, 4 of 4)

- [Motion Segmentation for Neuromorphic Aerial Surveillance](https://mlsanity.com/paper/2405.15209/) (2024)
- [Iterative Event-based Motion Segmentation by Variational Contrast Maximization](https://mlsanity.com/paper/2504.18447/) (2025)
- [Rethinking Event-Based Object Detection through Representation-Level Temporal Aggregation and Model-Level Hypergraph Reasoning](https://mlsanity.com/paper/2605.08825/) (2026)
- [EV-LayerSegNet: Self-supervised Motion Segmentation using Event Cameras](https://mlsanity.com/paper/2506.06596/) (2025)

## Abstract (arXiv)

Moving object segmentation is critical to interpret scene dynamics for robotic navigation systems in challenging environments. Neuromorphic vision sensors are tailored for motion perception due to their asynchronous nature, high temporal resolution, and reduced power consumption. However, their unconventional output requires novel perception paradigms to leverage their spatially sparse and temporally dense nature. In this work, we propose a novel event-based motion segmentation algorithm using a Graph Transformer Neural Network, dubbed GTNN. Our proposed algorithm processes event streams as 3D graphs by a series of nonlinear transformations to unveil local and global spatiotemporal correlations between events. Based on these correlations, events belonging to moving objects are segmented from the background without prior knowledge of the dynamic scene geometry. The algorithm is trained on publicly available datasets including MOD, EV-IMO, and textcolorblackEV-IMO2 using the proposed training scheme to facilitate efficient training on extensive datasets. Moreover, we introduce the Dynamic Object Mask-aware Event Labeling (DOMEL) approach for generating approximate ground-truth labels for event-based motion segmentation datasets. We use DOMEL to label our own recorded Event dataset for Motion Segmentation (EMS-DOMEL), which we release to the public for further research and benchmarking. Rigorous experiments are conducted on several unseen publicly-available datasets where the results revealed that GTNN outperforms state-of-the-art methods in the presence of dynamic background variations, motion patterns, and multiple dynamic objects with varying sizes and velocities. GTNN achieves significant performance gains with an average increase of 9.4% and 4.5% in terms of motion segmentation accuracy (IoU%) and detection rate (DR%), respectively.