# A Neuromorphic Architecture for Reinforcement Learning from Real-Valued Observations

arXiv: https://arxiv.org/abs/2307.02947 · HTML: https://mlsanity.com/paper/2307.02947/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 46 references that are in the map.
- First posted on arXiv 6 Jul 2023.

## Builds on (most cited first, 13 of 13)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Eligibility Traces and Plasticity on Behavioral Time Scales: Experimental Support of neoHebbian Three-Factor Learning Rules](https://mlsanity.com/paper/1801.05219/) (2018)
- [HFirst: A Temporal Approach to Object Recognition](https://mlsanity.com/paper/1508.01176/) (2015)
- [Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control](https://mlsanity.com/paper/2010.09635/) (2020)
- [Event-based Feature Extraction Using Adaptive Selection Thresholds](https://mlsanity.com/paper/1907.07853/) (2019)
- [First-spike based visual categorization using reward-modulated STDP](https://mlsanity.com/paper/1705.09132/) (2017)
- [An optimised deep spiking neural network architecture without gradients](https://mlsanity.com/paper/2109.12813/) (2021)
- [Event-driven Spectrotemporal Feature Extraction and Classification using a Silicon Cochlea Model](https://mlsanity.com/paper/2212.07136/) (2022)

## Built on by (most cited first, 2 of 2)

- [HyperSNN: A new efficient and robust deep learning model for resource constrained control applications](https://mlsanity.com/paper/2308.08222/) (2023)
- [All On-Board: Fully On-Chip Neuromorphic Q-Learning with Embedded CartPole Simulation](https://mlsanity.com/paper/2609.32317/) (2026)

## Abstract (arXiv)

Reinforcement Learning (RL) provides a powerful framework for decision-making in complex environments. However, implementing RL in hardware-efficient and bio-inspired ways remains a challenge. This paper presents a novel Spiking Neural Network (SNN) architecture for solving RL problems with real-valued observations. The proposed model incorporates multi-layered event-based clustering, with the addition of Temporal Difference (TD)-error modulation and eligibility traces, building upon prior work. An ablation study confirms the significant impact of these components on the proposed model's performance. A tabular actor-critic algorithm with eligibility traces and a state-of-the-art Proximal Policy Optimization (PPO) algorithm are used as benchmarks. Our network consistently outperforms the tabular approach and successfully discovers stable control policies on classic RL environments: mountain car, cart-pole, and acrobot. The proposed model offers an appealing trade-off in terms of computational and hardware implementation requirements. The model does not require an external memory buffer nor a global error gradient computation, and synaptic updates occur online, driven by local learning rules and a broadcasted TD-error signal. Thus, this work contributes to the development of more hardware-efficient RL solutions.