mlsanityData to 7 Oct 2026

OffCon3: What is state of the art anyway?

arXiv 2101.11331 ↗First posted 27 Jan 2021cs.LG math.OCcited by 5

Not on the map. Why?

Abstract (arXiv)

Two popular approaches to model-free continuous control tasks are SAC and TD3. At first glance these approaches seem rather different; SAC aims to solve the entropy-augmented MDP by minimising the KL-divergence between a stochastic proposal policy and a hypotheical energy-basd soft Q-function policy, whereas TD3 is derived from DPG, which uses a deterministic policy to perform policy gradient ascent along the value function. In reality, both approaches are remarkably similar, and belong to a family of approaches we call `Off-Policy Continuous Generalized Policy Iteration'. This illuminates their similar performance in most continuous control benchmarks, and indeed when hyperparameters are matched, their performance can be statistically indistinguishable. To further remove any difference due to implementation, we provide OffCon3 (Off-Policy Continuous Control: Consolidated), a code base featuring state-of-the-art versions of both algorithms.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 15 of its 19 references that are in the map. First posted on arXiv 27 Jan 2021.

Builds on

15 of 19 references.

References · 15

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2013Auto-Encoding Variational Bayescited by 3,980
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  7. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  8. 2014Convex Optimization: Algorithms and Complexitycited by 569
  9. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  10. 2019Monte Carlo Gradient Estimation in Machine Learningcited by 281
  11. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 178
  12. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
3 more
  1. 2020Model-Augmented Actor-Critic: Backpropagating through Pathscited by 61
  2. 2019Soft Policy Gradient Method for Maximum Entropy Deep Reinforcement Learningcited by 18
  3. 2020Variational Model-based Policy Optimizationcited by 7

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 0; 2023: 2; 2024: 1; 2025: 1.

5 papers.

  1. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  2. 2021Tactical Optimism and Pessimism for Deep Reinforcement Learningcited by 55
  3. 2023Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Functioncited by 19
  4. 2025Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classificationcited by 19
  5. 2024AC4MPC: Actor-Critic Reinforcement Learning for Nonlinear Model Predictive Controlcited by 4