# Synthesizing Images from Spatio-Temporal Representations using Spike-based Backpropagation

arXiv: https://arxiv.org/abs/1906.08861 · HTML: https://mlsanity.com/paper/1906.08861/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2019.

## Built on by (most cited first, 5 of 5)

- [Underwater Image Enhancement by Convolutional Spiking Neural Networks](https://mlsanity.com/paper/2503.20485/) (2025)
- [Training Deep Spiking Auto-encoders without Bursting or Dying Neurons through Regularization](https://mlsanity.com/paper/2109.11045/) (2021)
- [Predictive Temporal Attention on Event-based Video Stream for Energy-efficient Situation Awareness](https://mlsanity.com/paper/2402.08936/) (2024)
- [Towards efficient keyword spotting using spike-based time difference encoders](https://mlsanity.com/paper/2503.15402/) (2025)
- [snnTrans-DHZ: A Lightweight Spiking Neural Network Architecture for Underwater Image Dehazing](https://mlsanity.com/paper/2504.11482/) (2025)

## Abstract (arXiv)

Spiking neural networks (SNNs) offer a promising alternative to current artificial neural networks to enable low-power event-driven neuromorphic hardware. Spike-based neuromorphic applications require processing and extracting meaningful information from spatio-temporal data, represented as series of spike trains over time. In this paper, we propose a method to synthesize images from multiple modalities in a spike-based environment. We use spiking auto-encoders to convert image and audio inputs into compact spatio-temporal representations that is then decoded for image synthesis. For this, we use a direct training algorithm that computes loss on the membrane potential of the output layer and back-propagates it by using a sigmoid approximation of the neuron's activation function to enable differentiability. The spiking autoencoders are benchmarked on MNIST and Fashion-MNIST and achieve very low reconstruction loss, comparable to ANNs. Then, spiking autoencoders are trained to learn meaningful spatio-temporal representations of the data, across the two modalities - audio and visual. We synthesize images from audio in a spike-based environment by first generating, and then utilizing such shared multi-modal spatio-temporal representations. Our audio to image synthesis model is tested on the task of converting TI-46 digits audio samples to MNIST images. We are able to synthesize images with high fidelity and the model achieves competitive performance against ANNs.