mlsanityData to 7 Oct 2026

SARC: Soft Actor Retrospective Critic

arXiv 2306.16503 ↗First posted 28 Jun 2023cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

The two-time scale nature of SAC, which is an actor-critic algorithm, is characterised by the fact that the critic estimate has not converged for the actor at any given time, but since the critic learns faster than the actor, it ensures eventual consistency between the two. Various strategies have been introduced in literature to learn better gradient estimates to help achieve better convergence. Since gradient estimates depend upon the critic, we posit that improving the critic can provide a better gradient estimate for the actor at each time. Utilizing this, we propose Soft Actor Retrospective Critic (SARC), where we augment the SAC critic loss with another loss term - retrospective loss - leading to faster critic convergence and consequently, better policy gradient estimates for the actor. An existing implementation of SAC can be easily adapted to SARC with minimal modifications. Through extensive experimentation and analysis, we show that SARC provides consistent improvement over SAC on benchmark environments. We plan to open-source the code and all experiment data at: https://github.com/sukritiverma1996/SARC.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 16 of its 27 references that are in the map. First posted on arXiv 28 Jun 2023.

Builds on

16 of 27 references.

References · 16

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  8. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  9. 2018DeepMind Control Suitecited by 935
  10. 2017Deep Reinforcement Learning that Matterscited by 727
  11. 2018Is Q-learning Provably Efficient?cited by 485
  12. 2016Sample Efficient Actor-Critic with Experience Replaycited by 185
4 more
  1. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 158
  2. 2020A Finite Time Analysis of Two Time-Scale Actor Critic Methodscited by 120
  3. 2017Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximationcited by 119
  4. 2020Retrospective Loss: Looking Back to Improve Training of Deep Neural Networkscited by 3

Built on by

Not cited yet.