mlsanityData to 7 Oct 2026

Variational Actor-Critic Algorithms

arXiv 2108.01215 ↗First posted 3 Aug 2021cs.LG math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce a class of variational actor-critic algorithms based on a variational formulation over both the value function and the policy. The objective function of the variational formulation consists of two parts: one for maximizing the value function and the other for minimizing the Bellman residual. Besides the vanilla gradient descent with both the value function and the policy updates, we propose two variants, the clipping method and the flipping method, in order to speed up the convergence. We also prove that, when the prefactor of the Bellman residual is sufficiently large, the fixed point of the algorithm is close to the optimal policy.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 17 of its 31 references that are in the map. First posted on arXiv 3 Aug 2021.

Builds on

17 of 31 references.

References · 17

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  7. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 258
  8. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  9. 2016Sample Efficient Actor-Critic with Experience Replaycited by 185
  10. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 174
  11. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 170
  12. 2017Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximationcited by 119
5 more
  1. 2017Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Controlcited by 92
  2. 2020Why resampling outperforms reweighting for correcting sampling bias with stochastic gradientscited by 34
  3. 2020Finite-Sample Analysis of Proximal Gradient TD Algorithmscited by 32
  4. 2019Importance Resampling for Off-policy Predictioncited by 26
  5. 2020Borrowing From the Future: Addressing Double Sampling in Model-free Controlcited by 2

Built on by

Not cited yet.