# MouseSIS: A Frames-and-Events Dataset for Space-Time Instance Segmentation of Mice

arXiv: https://arxiv.org/abs/2409.03358 · HTML: https://mlsanity.com/paper/2409.03358/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 52 references that are in the map.
- First posted on arXiv 5 Sep 2024.

## Builds on (most cited first, 30 of 41)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking](https://mlsanity.com/paper/1609.01775/) (2016)
- [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020)
- [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016)
- [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021)
- [XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model](https://mlsanity.com/paper/2207.07115/) (2022)
- [Video Instance Segmentation](https://mlsanity.com/paper/1905.04804/) (2019)
- [YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark](https://mlsanity.com/paper/1809.03327/) (2018)
- [MOT20: A benchmark for multi object tracking in crowded scenes](https://mlsanity.com/paper/2003.09003/) (2020)
- [High Speed and High Dynamic Range Video with an Event Camera](https://mlsanity.com/paper/1906.07165/) (2019)
- [MOTS: Multi-Object Tracking and Segmentation](https://mlsanity.com/paper/1902.03604/) (2019)
- [DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion](https://mlsanity.com/paper/2111.14690/) (2021)
- [Markerless tracking of user-defined features with deep learning](https://mlsanity.com/paper/1804.03142/) (2018)
- [Learning to Detect Objects with a 1 Megapixel Event Camera](https://mlsanity.com/paper/2009.13436/) (2020)
- [Need for Speed: A Benchmark for Higher Frame Rate Object Tracking](https://mlsanity.com/paper/1703.05884/) (2017)
- [MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking](https://mlsanity.com/paper/1504.01942/) (2015)
- [VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flows](https://mlsanity.com/paper/2108.05015/) (2021)
- [Event-based Moving Object Detection and Tracking](https://mlsanity.com/paper/1803.04523/) (2018)
- [Asynchronous, Photometric Feature Tracking using Events and Frames](https://mlsanity.com/paper/1807.09713/) (2018)
- [A Large Scale Event-based Detection Dataset for Automotive](https://mlsanity.com/paper/2001.08499/) (2020)
- [SeqFormer: Sequential Transformer for Video Instance Segmentation](https://mlsanity.com/paper/2112.08275/) (2021)
- [Event-aided Direct Sparse Odometry](https://mlsanity.com/paper/2204.07640/) (2022)
- [Secrets of Event-Based Optical Flow](https://mlsanity.com/paper/2207.10022/) (2022)
- [Object Tracking by Jointly Exploiting Frame and Event Domain](https://mlsanity.com/paper/2109.09052/) (2021)
- [Event Stream-based Visual Object Tracking: A High-Resolution Benchmark Dataset and A Novel Baseline](https://mlsanity.com/paper/2309.14611/) (2023)
- [BURST: A Benchmark for Unifying Object Recognition, Segmentation and Tracking in Video](https://mlsanity.com/paper/2209.12118/) (2022)

## Built on by (most cited first, 5 of 5)

- [EV-Flying: an Event-based Dataset for In-The-Wild Recognition of Flying Objects](https://mlsanity.com/paper/2506.04048/) (2025)
- [Segment Any Events with Language](https://mlsanity.com/paper/2601.23159/) (2026)
- [SIS-Challenge: Event-based Spatio-temporal Instance Segmentation Challenge at the CVPR 2025 Event-based Vision Workshop](https://mlsanity.com/paper/2508.12813/) (2025)
- [Reconstruction as a Bridge for Event-Based Visual Question Answering](https://mlsanity.com/paper/2512.11510/) (2025)
- [DIMOS: Disentangling Instance-level Moving Object Segmentation](https://mlsanity.com/paper/2606.12826/) (2026)

## Abstract (arXiv)

Enabled by large annotated datasets, tracking and segmentation of objects in videos has made remarkable progress in recent years. Despite these advancements, algorithms still struggle under degraded conditions and during fast movements. Event cameras are novel sensors with high temporal resolution and high dynamic range that offer promising advantages to address these challenges. However, annotated data for developing learning-based mask-level tracking algorithms with events is not available. To this end, we introduce: (i) a new task termed space-time instance segmentation, similar to video instance segmentation, whose goal is to segment instances throughout the entire duration of the sensor input (here, the input are quasi-continuous events and optionally aligned frames); and (ii) dname, a dataset for the new task, containing aligned grayscale frames and events. It includes annotated ground-truth labels (pixel-level instance segmentation masks) of a group of up to seven freely moving and interacting mice. We also provide two reference methods, which show that leveraging event data can consistently improve tracking performance, especially when used in combination with conventional cameras. The results highlight the potential of event-aided tracking in difficult scenarios. We hope our dataset opens the field of event-based video instance segmentation and enables the development of robust tracking algorithms for challenging conditions.urlhttps://github.com/tub-rip/MouseSIS