# Bellman Memory Units: A neuromorphic framework for synaptic reinforcement learning with an evolving network topology

arXiv: https://arxiv.org/abs/2511.16066 · HTML: https://mlsanity.com/paper/2511.16066/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Spiking and neuromorphic RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 7 of its 20 references that are in the map.
- First posted on arXiv 20 Nov 2025.

## Placed in

- Deep reinforcement learning methods / [Spiking and neuromorphic RL](https://mlsanity.com/deeprl/spiking-and-neuromorphic-rl/)

## Builds on (most cited first, 7 of 7)

- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Continual Reinforcement Learning with Complex Synapses](https://mlsanity.com/paper/1802.07239/) (2018)
- [Nengo and low-power AI hardware for robust, embedded neurorobotics](https://mlsanity.com/paper/2007.10227/) (2020)
- [Optimizing with Low Budgets: a Comparison on the Black-box Optimization Benchmarking Suite and OpenAI Gym](https://mlsanity.com/paper/2310.00077/) (2023)
- [Gradient-Free Neural Network Training via Synaptic-Level Reinforcement Learning](https://mlsanity.com/paper/2105.14383/) (2021)
- [Continual Learning with Hebbian Plasticity in Sparse and Predictive Coding Networks: A Survey and Perspective](https://mlsanity.com/paper/2407.17305/) (2024)
- [Evolving Spiking Neural Networks for Nonlinear Control Problems](https://mlsanity.com/paper/1903.01180/) (2019)

## Abstract (arXiv)

Application of neuromorphic edge devices for control is limited by the constraints on gradient-free online learning and scalability of the hardware across control problems. This paper introduces a synaptic Q-learning algorithm for the control of the classical Cartpole, where the Bellman equations are incorporated at the synaptic level. This formulation enables the iterative evolution of the network topology, represented as a directed graph, throughout the training process. This is followed by a similar approach called neuromorphic Bellman Memory Units (BMU(s)), which are implemented with the Neural Engineering Framework on Intel's Loihi neuromorphic chip. Topology evolution, in conjunction with mixed-signal computation, leverages the optimization of the number of neurons and synapses that could be used to design spike-based reinforcement learning accelerators. The proposed architecture can potentially reduce resource utilization on board, aiding the manufacturing of compact application-specific neuromorphic ICs. Moreover, the on-chip learning introduced in this work and implemented on a neuromorphic chip can enable adaptation to unseen control scenarios.