# GRAC: Self-Guided and Self-Regularized Actor-Critic

arXiv: https://arxiv.org/abs/2009.08973 · HTML: https://mlsanity.com/paper/2009.08973/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Sep 2020.

## Built on by (most cited first, 21 of 21)

- [Temporal Difference Learning for Model Predictive Control](https://mlsanity.com/paper/2203.04955/) (2022)
- [Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone](https://mlsanity.com/paper/2412.06685/) (2024)
- [Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithms](https://mlsanity.com/paper/2401.11963/) (2024)
- [Combining Evolution and Deep Reinforcement Learning for Policy Search: a Survey](https://mlsanity.com/paper/2203.14009/) (2022)
- [Stabilizing Reinforcement Learning in Differentiable Multiphysics Simulation](https://mlsanity.com/paper/2412.12089/) (2024)
- [Functional Regularization for Reinforcement Learning via Learned Fourier Features](https://mlsanity.com/paper/2112.03257/) (2021)
- [Learning Dynamics and Generalization in Reinforcement Learning](https://mlsanity.com/paper/2206.02126/) (2022)
- [ERL-Re2: Efficient Evolutionary Reinforcement Learning with Shared State Representation and Individual Policy Representation](https://mlsanity.com/paper/2210.17375/) (2022)
- [Bridging the Gap Between Target Networks and Functional Regularization](https://mlsanity.com/paper/2106.02613/) (2021)
- [Evolutionary Action Selection for Gradient-based Policy Learning](https://mlsanity.com/paper/2201.04286/) (2022)
- [Adaptively Calibrated Critic Estimates for Deep Reinforcement Learning](https://mlsanity.com/paper/2111.12673/) (2021)
- [Towards Automatic Actor-Critic Solutions to Continuous Control](https://mlsanity.com/paper/2106.08918/) (2021)
- [Evolutionary Reinforcement Learning: A Systematic Review and Future Directions](https://mlsanity.com/paper/2402.13296/) (2024)
- [Bridging the Performance Gap Between Target-Free and Target-Based Reinforcement Learning](https://mlsanity.com/paper/2506.04398/) (2025)
- [Mitigating Suboptimality of Deterministic Policy Gradients in Complex Q-functions](https://mlsanity.com/paper/2410.11833/) (2024)
- [Use the Online Network If You Can: Towards Fast and Stable Reinforcement Learning](https://mlsanity.com/paper/2510.02590/) (2025)
- [Temporal Difference Learning with Constrained Initial Representations](https://mlsanity.com/paper/2602.11800/) (2026)
- [Towards Exploiting Geometry and Time for Fast Off-Distribution Adaptation in Multi-Task Robot Learning](https://mlsanity.com/paper/2106.13237/) (2021)
- [Bridging the Gap Between Target Networks and Functional Regularization](https://mlsanity.com/paper/2210.12282/) (2022)
- [Actor-Free Continuous Control via Structurally Maximizable Q-Functions](https://mlsanity.com/paper/2510.18828/) (2025)
- [Multi-Agent Cross-Entropy Method with Monotonic Nonlinear Critic Decomposition](https://mlsanity.com/paper/2511.18671/) (2025)

## Abstract (arXiv)

Deep reinforcement learning (DRL) algorithms have successfully been demonstrated on a range of challenging decision making and control tasks. One dominant component of recent deep reinforcement learning algorithms is the target network which mitigates the divergence when learning the Q function. However, target networks can slow down the learning process due to delayed function updates. Our main contribution in this work is a self-regularized TD-learning method to address divergence without requiring a target network. Additionally, we propose a self-guided policy improvement method by combining policy-gradient with zero-order optimization to search for actions associated with higher Q-values in a broad neighborhood. This makes learning more robust to local noise in the Q function approximation and guides the updates of our actor network. Taken together, these components define GRAC, a novel self-guided and self-regularized actor critic algorithm. We evaluate GRAC on the suite of OpenAI gym tasks, achieving or outperforming state of the art in every environment tested.