mlsanityData to 7 Oct 2026

Temporal-Difference Learning Using Distributed Error Signals

arXiv 2411.03604 ↗First posted 6 Nov 2024cs.LG cs.NEcited by 3

Not on the map. Why?

Abstract (arXiv)

A computational problem in biological reward-based learning is how credit assignment is performed in the nucleus accumbens (NAc). Much research suggests that NAc dopamine encodes temporal-difference (TD) errors for learning value predictions. However, dopamine is synchronously distributed in regionally homogeneous concentrations, which does not support explicit credit assignment (like used by backpropagation). It is unclear whether distributed errors alone are sufficient for synapses to make coordinated updates to learn complex, nonlinear reward-based learning tasks. We design a new deep Q-learning algorithm, Artificial Dopamine, to computationally demonstrate that synchronously distributed, per-layer TD errors may be sufficient to learn surprisingly complex RL tasks. We empirically evaluate our algorithm on MinAtar, the DeepMind Control Suite, and classic control tasks, and show it often achieves comparable performance to deep RL algorithms that use backpropagation.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 26 of its 69 references that are in the map. First posted on arXiv 6 Nov 2024.

Builds on

26 of 69 references.

References · 26

  1. 2016Layer Normalizationcited by 5,579
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  6. 2018DeepMind Control Suitecited by 935
  7. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  8. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  9. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  10. 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 262
  11. 2014Difference Target Propagationcited by 144
  12. 2019Deep Learning without Weight Transportcited by 87
14 more
  1. 2020Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Futurecited by 83
  2. 2020Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Researchcited by 78
  3. 2016Towards deep learning with segregated dendritescited by 71
  4. 2022Error-driven Input Modulation: Solving the Credit Assignment Problem without a Backward Passcited by 63
  5. 2022Hebbian Deep Learning Without Feedbackcited by 61
  6. 2023Empirical Design in Reinforcement Learningcited by 54
  7. 2020The Neural Coding Framework for Learning Generative Modelscited by 44
  8. 2020Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticitycited by 39
  9. 2021Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policiescited by 37
  10. 2020Deep Reinforcement Learning and its Neuroscientific Implicationscited by 33
  11. 2020Two Routes to Scalable Credit Assignment without Weight Symmetrycited by 19
  12. 2023Brain-Inspired Machine Intelligence: A Survey of Neurobiologically-Plausible Credit Assignmentcited by 13
  13. 2021Backprop-Free Reinforcement Learning with Active Neural Generative Codingcited by 12
  14. 2022Neuromorphic Artificial Intelligence Systemscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Local Pairwise Distance Matching for Backpropagation-Free Reinforcement Learningcited by 0
  2. 2025Local Reinforcement Learning with Action-Conditioned Root Mean Squared Q-Functionscited by 0
  3. 2026From Passive Reuse to Active Reasoning: Grounding Large Language Models for Neuro-Symbolic Experience Replaycited by 0