# Reinforcement Learning with Feedback-modulated TD-STDP

arXiv: https://arxiv.org/abs/2008.13044 · HTML: https://mlsanity.com/paper/2008.13044/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Aug 2020.

## Built on by (most cited first, 4 of 4)

- [Augmenting learning in neuro-embodied systems through neurobiological first principles](https://mlsanity.com/paper/2407.04525/) (2024)
- [Improving the adaptive and continuous learning capabilities of artificial neural networks: Lessons from multi-neuromodulatory dynamics](https://mlsanity.com/paper/2501.06762/) (2025)
- [Biologically Plausible Variational Policy Gradient with Spiking Recurrent Winner-Take-All Networks](https://mlsanity.com/paper/2210.13225/) (2022)
- [Learning to Control Dynamical Agents via Spiking Neural Networks and Metropolis-Hastings Sampling](https://mlsanity.com/paper/2507.09540/) (2025)

## Abstract (arXiv)

Spiking neuron networks have been used successfully to solve simple reinforcement learning tasks with continuous action set applying learning rules based on spike-timing-dependent plasticity (STDP). However, most of these models cannot be applied to reinforcement learning tasks with discrete action set since they assume that the selected action is a deterministic function of firing rate of neurons, which is continuous. In this paper, we propose a new STDP-based learning rule for spiking neuron networks which contains feedback modulation. We show that the STDP-based learning rule can be used to solve reinforcement learning tasks with discrete action set at a speed similar to standard reinforcement learning algorithms when applied to the CartPole and LunarLander tasks. Moreover, we demonstrate that the agent is unable to solve these tasks if feedback modulation is omitted from the learning rule. We conclude that feedback modulation allows better credit assignment when only the units contributing to the executed action and TD error participate in learning.