# Biologically Plausible Variational Policy Gradient with Spiking Recurrent Winner-Take-All Networks

arXiv: https://arxiv.org/abs/2210.13225 · HTML: https://mlsanity.com/paper/2210.13225/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 37 references that are in the map.
- First posted on arXiv 21 Oct 2022.

## Builds on (most cited first, 12 of 12)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation](https://mlsanity.com/paper/1602.05179/) (2016)
- [SuperSpike: Supervised learning in multi-layer spiking neural networks](https://mlsanity.com/paper/1705.11146/) (2017)
- [A Comprehensive Analysis on Adversarial Robustness of Spiking Neural Networks](https://mlsanity.com/paper/1905.02704/) (2019)
- [Demonstrating Advantages of Neuromorphic Computation: A Pilot Study](https://mlsanity.com/paper/1811.03618/) (2018)
- [VOWEL: A Local Online Learning Rule for Recurrent Networks of Probabilistic Spiking Winner-Take-All Circuits](https://mlsanity.com/paper/2004.09416/) (2020)
- [Towards the Next Generation of Retinal Neuroprosthesis: Visual Computation with Spikes](https://mlsanity.com/paper/2001.04064/) (2020)
- [Reinforcement Learning with Feedback-modulated TD-STDP](https://mlsanity.com/paper/2008.13044/) (2020)
- [Coarse scale representation of spiking neural networks: backpropagation through spikes and application to neuromorphic hardware](https://mlsanity.com/paper/2007.06176/) (2020)

## Abstract (arXiv)

One stream of reinforcement learning research is exploring biologically plausible models and algorithms to simulate biological intelligence and fit neuromorphic hardware. Among them, reward-modulated spike-timing-dependent plasticity (R-STDP) is a recent branch with good potential in energy efficiency. However, current R-STDP methods rely on heuristic designs of local learning rules, thus requiring task-specific expert knowledge. In this paper, we consider a spiking recurrent winner-take-all network, and propose a new R-STDP method, spiking variational policy gradient (SVPG), whose local learning rules are derived from the global policy gradient and thus eliminate the need for heuristic designs. In experiments of MNIST classification and Gym InvertedPendulum, our SVPG achieves good training performance, and also presents better robustness to various kinds of noises than conventional methods.