# Backprop-Free Reinforcement Learning with Active Neural Generative Coding

arXiv: https://arxiv.org/abs/2107.07046 · HTML: https://mlsanity.com/paper/2107.07046/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 105 references that are in the map.
- First posted on arXiv 10 Jul 2021.

## Builds on (most cited first, 30 of 36)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [Planning to Explore via Self-Supervised World Models](https://mlsanity.com/paper/2005.05960/) (2020)
- [Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning](https://mlsanity.com/paper/1708.02596/) (2017)
- [Large-Scale Study of Curiosity-Driven Learning](https://mlsanity.com/paper/1808.04355/) (2018)
- [Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation](https://mlsanity.com/paper/1602.05179/) (2016)
- [Efficient Exploration via State Marginal Matching](https://mlsanity.com/paper/1906.05274/) (2019)
- [Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning](https://mlsanity.com/paper/1712.06567/) (2017)
- [Difference Target Propagation](https://mlsanity.com/paper/1412.7525/) (2014)
- [Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning](https://mlsanity.com/paper/1509.08731/) (2015)
- [Decoupled Neural Interfaces using Synthetic Gradients](https://mlsanity.com/paper/1608.05343/) (2016)
- [Model-Based Active Exploration](https://mlsanity.com/paper/1810.12162/) (2018)
- [Iterative Amortized Inference](https://mlsanity.com/paper/1807.09356/) (2018)
- [The Uncertainty Bellman Equation and Exploration](https://mlsanity.com/paper/1709.05380/) (2017)
- [Towards deep learning with segregated dendrites](https://mlsanity.com/paper/1610.00161/) (2016)
- [Towards Biologically Plausible Deep Learning](https://mlsanity.com/paper/1502.04156/) (2015)
- [Deep Active Inference as Variational Policy Gradients](https://mlsanity.com/paper/1907.03876/) (2019)
- [Reinforcement Learning through Active Inference](https://mlsanity.com/paper/2002.12636/) (2020)
- [Meta-Learning through Hebbian Plasticity in Random Networks](https://mlsanity.com/paper/2007.02686/) (2020)
- [Deep Active Inference](https://mlsanity.com/paper/1709.02341/) (2017)
- [The Neural Coding Framework for Learning Generative Models](https://mlsanity.com/paper/2012.03405/) (2020)
- [Biologically Motivated Algorithms for Propagating Local Target Representations](https://mlsanity.com/paper/1805.11703/) (2018)
- [Continual Learning of Recurrent Neural Networks by Locally Aligning Distributed Representations](https://mlsanity.com/paper/1810.07411/) (2018)

## Built on by (most cited first, 12 of 12)

- [Brain-inspired Computational Intelligence via Predictive Coding](https://mlsanity.com/paper/2308.07870/) (2023)
- [Predictive Coding beyond Gaussian Distributions](https://mlsanity.com/paper/2211.03481/) (2022)
- [Is Bio-Inspired Learning Better than Backprop? Benchmarking Bio Learning vs. Backprop](https://mlsanity.com/paper/2212.04614/) (2022)
- [Predictive Coding Networks and Inference Learning: Tutorial and Survey](https://mlsanity.com/paper/2407.04117/) (2024)
- [Active Predicting Coding: Brain-Inspired Reinforcement Learning for Sparse Reward Robotic Control Problems](https://mlsanity.com/paper/2209.09174/) (2022)
- [Active Sensing with Predictive Coding and Uncertainty Minimization](https://mlsanity.com/paper/2307.00668/) (2023)
- [Maze Learning using a Hyperdimensional Predictive Processing Cognitive Architecture](https://mlsanity.com/paper/2204.00619/) (2022)
- [Temporal-Difference Learning Using Distributed Error Signals](https://mlsanity.com/paper/2411.03604/) (2024)
- [A Neuro-mimetic Realization of the Common Model of Cognition via Hebbian Learning and Free Energy Minimization](https://mlsanity.com/paper/2310.15177/) (2023)
- [Active Inference in Hebbian Learning Networks](https://mlsanity.com/paper/2306.05053/) (2023)
- [Local Pairwise Distance Matching for Backpropagation-Free Reinforcement Learning](https://mlsanity.com/paper/2507.11367/) (2025)
- [Optimizing Neurorobot Policy under Limited Demonstration Data through Preference Regret](https://mlsanity.com/paper/2604.03523/) (2026)

## Abstract (arXiv)

In humans, perceptual awareness facilitates the fast recognition and extraction of information from sensory input. This awareness largely depends on how the human agent interacts with the environment. In this work, we propose active neural generative coding, a computational framework for learning action-driven generative models without backpropagation of errors (backprop) in dynamic environments. Specifically, we develop an intelligent agent that operates even with sparse rewards, drawing inspiration from the cognitive theory of planning as inference. We demonstrate on several simple control problems that our framework performs competitively with deep Q-learning. The robust performance of our agent offers promising evidence that a backprop-free approach for neural inference and learning can drive goal-directed behavior.