mlsanityData to 14 Oct 2026

Bellman Memory Units: A neuromorphic framework for synaptic reinforcement learning with an evolving network topology

arXiv 2511.16066 ↗First posted 20 Nov 2025eess.SY cs.NE cs.SYcited by 0

On the map: Deep reinforcement learning methods → Spiking and neuromorphic RL

Abstract (arXiv)

Application of neuromorphic edge devices for control is limited by the constraints on gradient-free online learning and scalability of the hardware across control problems. This paper introduces a synaptic Q-learning algorithm for the control of the classical Cartpole, where the Bellman equations are incorporated at the synaptic level. This formulation enables the iterative evolution of the network topology, represented as a directed graph, throughout the training process. This is followed by a similar approach called neuromorphic Bellman Memory Units (BMU(s)), which are implemented with the Neural Engineering Framework on Intel's Loihi neuromorphic chip. Topology evolution, in conjunction with mixed-signal computation, leverages the optimization of the number of neurons and synapses that could be used to design spike-based reinforcement learning accelerators. The proposed architecture can potentially reduce resource utilization on board, aiding the manufacturing of compact application-specific neuromorphic ICs. Moreover, the on-chip learning introduced in this work and implemented on a neuromorphic chip can enable adaptation to unseen control scenarios.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Spiking and neuromorphic RL, one of 15 directions in Deep reinforcement learning methods. Builds on 7 of its 20 references that are in the map. First posted on arXiv 20 Nov 2025.

Builds on

7 of 20 references.

References · 7

  1. 2019Event-based Vision: A Surveycited by 1,016
  2. 2018Continual Reinforcement Learning with Complex Synapsescited by 32
  3. 2020Nengo and low-power AI hardware for robust, embedded neuroroboticscited by 15
  4. 2023Optimizing with Low Budgets: a Comparison on the Black-box Optimization Benchmarking Suite and OpenAI Gymcited by 4
  5. 2021Gradient-Free Neural Network Training via Synaptic-Level Reinforcement Learningcited by 3
  6. 2024Continual Learning with Hebbian Plasticity in Sparse and Predictive Coding Networks: A Survey and Perspectivecited by 3
  7. 2019Evolving Spiking Neural Networks for Nonlinear Control Problemscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026All On-Board: Fully On-Chip Neuromorphic Q-Learning with Embedded CartPole Simulation
  2. 2025Hardware-Aware Fine-Tuning of Spiking Q-Networks on the SpiNNaker2 Neuromorphic Platform
  3. 2024Self-training superconducting neuromorphic circuits using reinforcement learning rules
  4. 2026Brain-inspired AI for Edge Intelligence: a systematic review
  5. 2025Enabling Efficient Processing of Spiking Neural Networks with On-Chip Learning on Commodity Neuromorphic Processors for Edge AI Systems
  6. 2024Neuromorphic force-control in an industrial task: validating energy and latency benefits ↗
  7. 2023CIMulator: A Comprehensive Simulation Platform for Computing-In-Memory Circuit Macros with Low Bit-Width and Real Memory Materials
  8. 2025Autonomous Reinforcement Learning Robot Control with Intel's Loihi 2 Neuromorphic Hardware