mlsanityData to 7 Oct 2026

Active Predicting Coding: Brain-Inspired Reinforcement Learning for Sparse Reward Robotic Control Problems

arXiv 2209.09174 ↗First posted 19 Sep 2022cs.RO cs.LG cs.NEcited by 6

Not on the map. Why?

Abstract (arXiv)

In this article, we propose a backpropagation-free approach to robotic control through the neuro-cognitive computational framework of neural generative coding (NGC), designing an agent built completely from powerful predictive coding/processing circuits that facilitate dynamic, online learning from sparse rewards, embodying the principles of planning-as-inference. Concretely, we craft an adaptive agent system, which we call active predictive coding (ActPC), that balances an internally-generated epistemic signal (meant to encourage intelligent exploration) with an internally-generated instrumental signal (meant to encourage goal-seeking behavior) to ultimately learn how to control various simulated robotic systems as well as a complex robotic arm using a realistic robotics simulator, i.e., the Surreal Robotics Suite, for the block lifting task and can pick-and-place problems. Notably, our experimental results demonstrate that our proposed ActPC agent performs well in the face of sparse (extrinsic) reward signals and is competitive with or outperforms several powerful backprop-based RL approaches.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 46 of its 102 references that are in the map. First posted on arXiv 19 Sep 2022.

Builds on

46 of 102 references.

References · 46

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  7. 2016Generative Adversarial Imitation Learningcited by 1,576
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  9. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  10. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  11. 2017Hindsight Experience Replaycited by 1,096
  12. 2018Exploration by Random Network Distillationcited by 966
34 more
  1. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  2. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  3. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  4. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  5. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  6. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  8. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  9. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  10. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  11. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  12. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
  13. 2016Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagationcited by 251
  14. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
  15. 2019Efficient Exploration via State Marginal Matchingcited by 186
  16. 2014Difference Target Propagationcited by 144
  17. 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 142
  18. 2016Decoupled Neural Interfaces using Synthetic Gradientscited by 135
  19. 2018Self-Imitation Learningcited by 135
  20. 2018Model-Based Active Explorationcited by 105
  21. 2017A Deeper Look at Experience Replaycited by 104
  22. 2017The Uncertainty Bellman Equation and Explorationcited by 81
  23. 2016Towards deep learning with segregated dendritescited by 71
  24. 2022Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstrationcited by 54
  25. 2019Deep Active Inference as Variational Policy Gradientscited by 52
  26. 2020Meta-Learning through Hebbian Plasticity in Random Networkscited by 47
  27. 2017Deep Active Inferencecited by 44
  28. 2020The Neural Coding Framework for Learning Generative Modelscited by 44
  29. 2018Biologically Motivated Algorithms for Propagating Local Target Representationscited by 43
  30. 2018Continual Learning of Recurrent Neural Networks by Locally Aligning Distributed Representationscited by 32
  31. 2019Spiking Neural Predictive Coding for Continual Learning from Data Streamscited by 17
  32. 2019A Unified Bellman Optimality Principle Combining Reward Maximization and Empowermentcited by 15
  33. 2021Backprop-Free Reinforcement Learning with Active Neural Generative Codingcited by 12
  34. 2017Training Language Models Using Target-Propagationcited by 5

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 2; 2024: 2.

6 papers.

  1. 2023Brain-inspired Computational Intelligence via Predictive Codingcited by 30
  2. 2022Predictive Coding beyond Gaussian Distributionscited by 16
  3. 2023Real-Time Recurrent Reinforcement Learningcited by 6
  4. 2022Robust Graph Representation Learning via Predictive Codingcited by 5
  5. 2024Deep hybrid models: infer and plan in a dynamic worldcited by 0
  6. 2024Dynamic planning in hierarchical active inferencecited by 0