# Encoding Event-Based Data With a Hybrid SNN Guided Variational Auto-encoder in Neuromorphic Hardware

arXiv: https://arxiv.org/abs/2104.00165 · HTML: https://mlsanity.com/paper/2104.00165/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 30 references that are in the map.
- First posted on arXiv 31 Mar 2021.

## Builds on (most cited first, 13 of 13)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [Autoencoding beyond pixels using a learned similarity metric](https://mlsanity.com/paper/1512.09300/) (2015)
- [Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccades](https://mlsanity.com/paper/1507.07629/) (2015)
- [SLAYER: Spike Layer Error Reassignment in Time](https://mlsanity.com/paper/1810.08646/) (2018)
- [SuperSpike: Supervised learning in multi-layer spiking neural networks](https://mlsanity.com/paper/1705.11146/) (2017)
- [Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks](https://mlsanity.com/paper/2003.06696/) (2020)
- [ClusterGAN : Latent Space Clustering in Generative Adversarial Networks](https://mlsanity.com/paper/1809.03627/) (2018)
- [Guided Variational Autoencoder for Disentanglement Learning](https://mlsanity.com/paper/2004.01255/) (2020)
- [Brain-Inspired Learning on Neuromorphic Substrates](https://mlsanity.com/paper/2010.11931/) (2020)
- [Biologically inspired alternatives to backpropagation through time for learning in recurrent neural nets](https://mlsanity.com/paper/1901.09049/) (2019)
- [Online Few-shot Gesture Learning on a Neuromorphic Processor](https://mlsanity.com/paper/2008.01151/) (2020)

## Built on by (most cited first, 5 of 5)

- [Spiking Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2306.17046/) (2023)
- [Spiking Diffusion Models](https://mlsanity.com/paper/2408.16467/) (2024)
- [ESVAE: An Efficient Spiking Variational Autoencoder with Reparameterizable Poisson Spiking Sampling](https://mlsanity.com/paper/2310.14839/) (2023)
- [SteganoSNN: SNN-Based Audio-in-Image Steganography with Encryption](https://mlsanity.com/paper/2511.06573/) (2025)
- [Hybrid guided variational autoencoder for visual place recognition](https://mlsanity.com/paper/2601.09248/) (2026)

## Abstract (arXiv)

Neuromorphic hardware equipped with learning capabilities can adapt to new, real-time data. While models of Spiking Neural Networks (SNNs) can now be trained using gradient descent to reach an accuracy comparable to equivalent conventional neural networks, such learning often relies on external labels. However, real-world data is unlabeled which can make supervised methods inapplicable. To solve this problem, we propose a Hybrid Guided Variational Autoencoder (VAE) which encodes event based data sensed by a Dynamic Vision Sensor (DVS) into a latent space representation using an SNN. These representations can be used as an embedding to measure data similarity and predict labels in real-world data. We show that the Hybrid Guided-VAE achieves 87% classification accuracy on the DVSGesture dataset and it can encode the sparse, noisy inputs into an interpretable latent space representation, visualized through T-SNE plots. We also implement the encoder component of the model on neuromorphic hardware and discuss the potential for our algorithm to enable real-time learning from real-world event data.