# Efficient Gesture Recognition on Spiking Convolutional Networks Through Sensor Fusion of Event-Based and Depth Data

arXiv: https://arxiv.org/abs/2401.17064 · HTML: https://mlsanity.com/paper/2401.17064/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 28 references that are in the map.
- First posted on arXiv 30 Jan 2024.

## Builds on (most cited first, 8 of 8)

- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [SLAYER: Spike Layer Error Reassignment in Time](https://mlsanity.com/paper/1810.08646/) (2018)
- [Towards Biologically Plausible Deep Learning](https://mlsanity.com/paper/1502.04156/) (2015)
- [Bridging the Gap between Events and Frames through Unsupervised Domain Adaptation](https://mlsanity.com/paper/2109.02618/) (2021)
- [Online Few-shot Gesture Learning on a Neuromorphic Processor](https://mlsanity.com/paper/2008.01151/) (2020)
- [Deep SCNN-based Real-time Object Detection for Self-driving Vehicles Using LiDAR Temporal Data](https://mlsanity.com/paper/1912.07906/) (2019)
- [Reactive Neural Path Planning with Dynamic Obstacle Avoidance in a Condensed Configuration Space](https://mlsanity.com/paper/2207.03959/) (2022)

## Built on by (most cited first, 1 of 1)

- [SMV-EAR: Bring Spatiotemporal Multi-View Representation Learning into Efficient Event-Based Action Recognition](https://mlsanity.com/paper/2601.17391/) (2026)

## Abstract (arXiv)

As intelligent systems become increasingly important in our daily lives, new ways of interaction are needed. Classical user interfaces pose issues for the physically impaired and are partially not practical or convenient. Gesture recognition is an alternative, but often not reactive enough when conventional cameras are used. This work proposes a Spiking Convolutional Neural Network, processing event- and depth data for gesture recognition. The network is simulated using the open-source neuromorphic computing framework LAVA for offline training and evaluation on an embedded system. For the evaluation three open source data sets are used. Since these do not represent the applied bi-modality, a new data set with synchronized event- and depth data was recorded. The results show the viability of temporal encoding on depth information and modality fusion, even on differently encoded data, to be beneficial to network performance and generalization capabilities.