# Improving Performance of Spike-based Deep Q-Learning using Ternary Neurons

arXiv: https://arxiv.org/abs/2506.03392 · HTML: https://mlsanity.com/paper/2506.03392/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 60 references that are in the map.
- First posted on arXiv 3 Jun 2025.

## Builds on (most cited first, 26 of 26)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks](https://mlsanity.com/paper/1706.02609/) (2017)
- [Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05143/) (2016)
- [Going Deeper With Directly-Trained Larger Spiking Neural Networks](https://mlsanity.com/paper/2011.05280/) (2020)
- [Distributed Prioritized Experience Replay](https://mlsanity.com/paper/1803.00933/) (2018)
- [Training Spiking Neural Networks Using Lessons From Deep Learning](https://mlsanity.com/paper/2109.12894/) (2021)
- [Direct Training for Spiking Neural Networks: Faster, Larger, Better](https://mlsanity.com/paper/1809.05793/) (2018)
- [Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation](https://mlsanity.com/paper/2005.01807/) (2020)
- [Enabling Spike-based Backpropagation for Training Deep Neural Network Architectures](https://mlsanity.com/paper/1903.06379/) (2019)
- [A scalable multi-core architecture with heterogeneous memory structures for Dynamic Neuromorphic Asynchronous Processors (DYNAPs)](https://mlsanity.com/paper/1708.04198/) (2017)
- [Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control](https://mlsanity.com/paper/2010.09635/) (2020)
- [Ternary Spike: Learning Ternary Spikes for Spiking Neural Networks](https://mlsanity.com/paper/2312.06372/) (2023)
- [Reinforcement co-Learning of Deep and Spiking Neural Networks for Energy-Efficient Mapless Navigation with Neuromorphic Hardware](https://mlsanity.com/paper/2003.01157/) (2020)
- [SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanisms](https://mlsanity.com/paper/2406.03287/) (2024)
- [Human-Level Control through Directly-Trained Deep Spiking Q-Networks](https://mlsanity.com/paper/2201.07211/) (2021)
- [Strategy and Benchmark for Converting Deep Q-Networks to Event-Driven Spiking Neural Networks](https://mlsanity.com/paper/2009.14456/) (2020)
- [CLIF: Complementary Leaky Integrate-and-Fire Neuron for Spiking Neural Networks](https://mlsanity.com/paper/2402.04663/) (2024)
- [High-Performance Temporal Reversible Spiking Neural Networks with O(L) Training Memory and O(1) Inference Cost](https://mlsanity.com/paper/2405.16466/) (2024)
- [Solving the Spike Feature Information Vanishing Problem in Spiking Deep Q Network with Potential Based Normalization](https://mlsanity.com/paper/2206.03654/) (2022)
- [Spiking Convolutional Neural Networks for Text Classification](https://mlsanity.com/paper/2406.19230/) (2024)
- [A Comprehensive and Modularized Statistical Framework for Gradient Norm Equality in Deep Neural Networks](https://mlsanity.com/paper/2001.00254/) (2020)
- [Spiker: an FPGA-optimized Hardware acceleration for Spiking Neural Networks](https://mlsanity.com/paper/2201.06993/) (2022)
- [A Fully-Configurable Open-Source Software-Defined Digital Quantized Spiking Neural Core Architecture](https://mlsanity.com/paper/2404.02248/) (2024)
- [SF-DQN: Provable Knowledge Transfer using Successor Feature for Deep Reinforcement Learning](https://mlsanity.com/paper/2405.15920/) (2024)

## Built on by (most cited first, 3 of 3)

- [New Spiking Architecture for Multi-Modal Decision-Making in Autonomous Vehicles](https://mlsanity.com/paper/2512.01882/) (2025)
- [Efficient Aspect Term Extraction using Spiking Neural Network](https://mlsanity.com/paper/2601.06637/) (2026)
- [Fuzzy Encoding-Decoding to Improve Spiking Q-Learning Performance in Autonomous Driving](https://mlsanity.com/paper/2604.16436/) (2026)

## Abstract (arXiv)

We propose a new ternary spiking neuron model to improve the representation capacity of binary spiking neurons in deep Q-learning. Although a ternary neuron model has recently been introduced to overcome the limited representation capacity offered by binary spiking neurons, we show that its performance is worse than that of binary models in deep Q-learning tasks, contradicting previous findings from recent studies. Through mathematical and empirical analysis, we hypothesize that gradient estimation bias during training is the underlying cause. The proposed ternary spiking neuron model mitigates this issue by reducing the estimation bias. We use the proposed ternary spiking neuron as the fundamental computing unit in a deep spiking Q-learning network, which we call the deep asymmetric ternary spiking Q-network (DATSQN), and evaluate the network's performance in seven Atari games from the Gym environment. The results show that the proposed ternary spiking neuron mitigates the performance degradation of ternary neurons in DQN tasks and improves the mean game score relative to the binary baseline under the evaluation settings used in this paper.