# Deep Reinforcement Learning with Spiking Q-learning

arXiv: https://arxiv.org/abs/2201.09754 · HTML: https://mlsanity.com/paper/2201.09754/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 41 references that are in the map.
- First posted on arXiv 21 Jan 2022.

## Builds on (most cited first, 14 of 14)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks](https://mlsanity.com/paper/2007.05785/) (2020)
- [Deep Residual Learning in Spiking Neural Networks](https://mlsanity.com/paper/2102.04159/) (2021)
- [Training Deep Spiking Neural Networks using Backpropagation](https://mlsanity.com/paper/1608.08782/) (2016)
- [The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks](https://mlsanity.com/paper/1910.07407/) (2019)
- [Eligibility Traces and Plasticity on Behavioral Time Scales: Experimental Support of neoHebbian Three-Factor Learning Rules](https://mlsanity.com/paper/1801.05219/) (2018)
- [Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control](https://mlsanity.com/paper/2010.09635/) (2020)
- [Memory and information processing in neuromorphic systems](https://mlsanity.com/paper/1506.03264/) (2015)
- [Inherent Adversarial Robustness of Deep Spiking Neural Networks: Effects of Discrete Input Encoding and Non-Linear Activations](https://mlsanity.com/paper/2003.10399/) (2020)
- [A Comprehensive Analysis on Adversarial Robustness of Spiking Neural Networks](https://mlsanity.com/paper/1905.02704/) (2019)
- [Reinforcement co-Learning of Deep and Spiking Neural Networks for Energy-Efficient Mapless Navigation with Neuromorphic Hardware](https://mlsanity.com/paper/2003.01157/) (2020)
- [Visual Explanations from Spiking Neural Networks using Interspike Intervals](https://mlsanity.com/paper/2103.14441/) (2021)
- [Human-Level Control through Directly-Trained Deep Spiking Q-Networks](https://mlsanity.com/paper/2201.07211/) (2021)
- [Strategy and Benchmark for Converting Deep Q-Networks to Event-Driven Spiking Neural Networks](https://mlsanity.com/paper/2009.14456/) (2020)

## Built on by (most cited first, 18 of 18)

- [SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence](https://mlsanity.com/paper/2310.16620/) (2023)
- [Solving the Spike Feature Information Vanishing Problem in Spiking Deep Q Network with Potential Based Normalization](https://mlsanity.com/paper/2206.03654/) (2022)
- [Proxy Target: Bridging the Gap Between Discrete Spiking Neural Networks and Continuous Control](https://mlsanity.com/paper/2505.24161/) (2025)
- [A Low Latency Adaptive Coding Spiking Framework for Deep Reinforcement Learning](https://mlsanity.com/paper/2211.11760/) (2022)
- [Tuning Synaptic Connections instead of Weights by Genetic Algorithm in Spiking Policy Network](https://mlsanity.com/paper/2301.10292/) (2022)
- [Multi-compartment Neuron and Population Encoding Powered Spiking Neural Network for Deep Distributional Reinforcement Learning](https://mlsanity.com/paper/2301.07275/) (2023)
- [GRSN: Gated Recurrent Spiking Neurons for POMDPs and MARL](https://mlsanity.com/paper/2404.15597/) (2024)
- [CaRe-BN: Precise Moving Statistics for Stabilizing Spiking Neural Networks in Reinforcement Learning](https://mlsanity.com/paper/2509.23791/) (2025)
- [Enabling Energy-Efficient Simultaneous Multi-Task Reinforcement Learning through Spiking Neural Networks with Active Dendrites for Bio-inspired Generalist Agents](https://mlsanity.com/paper/2412.04847/) (2024)
- [Scalable Multi-Task Learning through Spiking Neural Networks with Adaptive Task-Switching Policy for Intelligent Autonomous Agents](https://mlsanity.com/paper/2504.13541/) (2025)
- [Adaptive Surrogate Gradients for Sequential Reinforcement Learning in Spiking Neural Networks](https://mlsanity.com/paper/2510.24461/) (2025)
- [Error Amplification Limits ANN-to-SNN Conversion in Continuous Control](https://mlsanity.com/paper/2601.21778/) (2026)
- [Energy-Efficient Visual Search by Eye Movement and Low-Latency Spiking Neural Network](https://mlsanity.com/paper/2310.06578/) (2023)
- [SpikingSoft: A Spiking Neuron Controller for Bio-inspired Locomotion with Soft Snake Robots](https://mlsanity.com/paper/2501.19072/) (2025)
- [Hardware-Aware Fine-Tuning of Spiking Q-Networks on the SpiNNaker2 Neuromorphic Platform](https://mlsanity.com/paper/2507.23562/) (2025)
- [Q-SpiRL: Quantum Spiking Reinforcement Learning for Adaptive Robot Navigation](https://mlsanity.com/paper/2605.20801/) (2026)
- [Spiking Neural Networks for Continuous Control: Neuromorphic Reinforcement Learning in Conventional Computing](https://mlsanity.com/paper/2608.22729/) (2026)
- [Common-Mode Errors Limit Low-Timestep Deep Spiking Q-Networks](https://mlsanity.com/paper/2610.07808/) (2026)

## Abstract (arXiv)

With the help of special neuromorphic hardware, spiking neural networks (SNNs) are expected to realize artificial intelligence (AI) with less energy consumption. It provides a promising energy-efficient way for realistic control tasks by combining SNNs with deep reinforcement learning (RL). There are only a few existing SNN-based RL methods at present. Most of them either lack generalization ability or employ Artificial Neural Networks (ANNs) to estimate value function in training. The former needs to tune numerous hyper-parameters for each scenario, and the latter limits the application of different types of RL algorithm and ignores the large energy consumption in training. To develop a robust spike-based RL method, we draw inspiration from non-spiking interneurons found in insects and propose the deep spiking Q-network (DSQN), using the membrane voltage of non-spiking neurons as the representation of Q-value, which can directly learn robust policies from high-dimensional sensory inputs using end-to-end RL. Experiments conducted on 17 Atari games demonstrate the DSQN is effective and even outperforms the ANN-based deep Q-network (DQN) in most games. Moreover, the experiments show superior learning stability and robustness to adversarial attacks of DSQN.