mlsanityData to 7 Oct 2026

Local Reinforcement Learning with Action-Conditioned Root Mean Squared Q-Functions

arXiv 2510.06649 ↗First posted 8 Oct 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

The Forward-Forward (FF) Algorithm is a recently proposed learning procedure for neural networks that employs two forward passes instead of the traditional forward and backward passes used in backpropagation. However, FF remains largely confined to supervised settings, leaving a gap at domains where learning signals can be yielded more naturally such as RL. In this work, inspired by FF's goodness function using layer activity statistics, we introduce Action-conditioned Root mean squared Q-Functions (ARQ), a novel value estimation method that applies a goodness function and action conditioning for local RL using temporal difference learning. Despite its simplicity and biological grounding, our approach achieves superior performance compared to state-of-the-art local backprop-free RL methods in the MinAtar and the DeepMind Control Suite benchmarks, while also outperforming algorithms trained with backpropagation on most tasks. Code can be found at https://github.com/agentic-learning-ai-lab/arq.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 51 of its 71 references that are in the map. First posted on arXiv 8 Oct 2025.

Builds on

51 of 71 references.

References · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2015Trust Region Policy Optimizationcited by 2,503
  7. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  8. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  9. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  10. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  11. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  12. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
38 more
  1. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  2. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  3. 2020Mastering Atari with Discrete World Modelscited by 990
  4. 2018DeepMind Control Suitecited by 935
  5. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  6. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  7. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  8. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  9. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  10. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  11. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  12. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  13. 2018Maximum a Posteriori Policy Optimisationcited by 318
  14. 2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 281
  15. 2017Noisy Networks for Explorationcited by 262
  16. 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 262
  17. 2016Direct Feedback Alignment Provides Learning in Deep Neural Networkscited by 240
  18. 2021Mastering Atari Games with Limited Datacited by 235
  19. 2019Training Neural Networks with Local Error Signalscited by 135
  20. 2017Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learningcited by 114
  21. 2018Greedy Layerwise Learning Can Scale to ImageNetcited by 113
  22. 2021V-Learning -- A Simple, Efficient, Decentralized Algorithm for Multiagent RLcited by 111
  23. 2022Gradients without Backpropagationcited by 73
  24. 2022Scaling Forward Gradient With Local Lossescited by 65
  25. 2021Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policiescited by 37
  26. 2017Lenient Multi-Agent Deep Reinforcement Learningcited by 35
  27. 2023The Predictive Forward-Forward Algorithmcited by 35
  28. 2015Decentralized Q-Learning for Stochastic Teams and Gamescited by 34
  29. 2023Can Forward Gradient Match Backpropagation?cited by 26
  30. 2023Convolutional Channel-wise Competitive Learning for the Forward-Forward Algorithmcited by 23
  31. 2022Solving Continuous Control via Q-learningcited by 21
  32. 2025μPC: Scaling Predictive Coding to 100+ Layer Networkscited by 14
  33. 2023Emergent representations in networks trained with the Forward-Forward algorithmcited by 11
  34. 2023Deep Model Predictive Optimizationcited by 7
  35. 2023How to guess a gradientcited by 7
  36. 2022MA2QL: A Minimalist Approach to Fully Decentralized Multi-Agent Reinforcement Learningcited by 5
  37. 2022Decentralized Policy Optimizationcited by 5
  38. 2023Best Possible Q-Learningcited by 4

50 most cited of 51 All 51 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Learning Value Functions from Undirected State-only Experience
  2. 2026Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities
  3. 2022Confidence-Conditioned Value Functions for Offline Reinforcement Learning
  4. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RL
  5. 2024Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning
  6. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
  7. 2026Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning
  8. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learning