mlsanityData to 14 Oct 2026

SACn: Soft Actor-Critic with n-step Returns

arXiv 2512.13165 ↗First posted 15 Dec 2025cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Soft Actor-Critic (SAC) is widely used in practical applications and is now one of the most relevant off-policy online model-free reinforcement learning (RL) methods. The technique of n-step returns is known to increase the convergence speed of RL algorithms compared to their 1-step returns-based versions. However, SAC is notoriously difficult to combine with n-step returns, since their usual combination introduces bias in off-policy algorithms due to the changes in action distribution. While this problem is solved by importance sampling, a method for estimating expected values of one distribution using samples from another distribution, importance sampling may result in numerical instability. In this work, we combine SAC with n-step returns in a way that overcomes this issue. We present an approach to applying numerically stable importance sampling with simplified hyperparameter selection. Furthermore, we analyze the entropy estimation approach of Soft Actor-Critic in the context of the n-step maximum entropy framework and formulate the τ-sampled entropy estimation to reduce the variance of the learning target. Finally, we formulate the Soft Actor-Critic with n-step returns (SACn) algorithm that we experimentally verify on MuJoCo simulated environments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 13 of its 35 references that are in the map. First posted on arXiv 15 Dec 2025.

Builds on

13 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Averaging n-step Returns Reduces Variance in Reinforcement Learning6 Feb

Other references · 12

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  6. 2015Prioritized Experience Replaycited by 1,099
  7. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  8. 2018Distributed Prioritized Experience Replaycited by 269
  9. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  10. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 195
  11. 2016Sample Efficient Actor-Critic with Experience Replaycited by 188
  12. 2021ACERAC: Efficient reinforcement learning in fine time discretizationcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Bidirectional Soft Actor-Critic: Leveraging Forward and Reverse KL Divergence for Efficient Reinforcement Learning
  2. 2025Max-Entropy Reinforcement Learning with Flow Matching and A Case Study on LQR
  3. 2025Effective Reinforcement Learning Control using Conservative Soft Actor-Critic
  4. 2026Off-Policy Actor-Critic with Sigmoid-Bounded Entropy for Real-World Robot Learning
  5. 2025DR-SAC: Distributionally Robust Soft Actor-Critic for Reinforcement Learning under Uncertainty
  6. 2024Frugal Actor-Critic: Sample Efficient Off-Policy Deep Reinforcement Learning Using Unique Experiences
  7. 2023Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning
  8. 2024Soft Actor-Critic with Beta Policy via Implicit Reparameterization Gradients