# Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Past

arXiv: https://arxiv.org/abs/1906.04009 · HTML: https://mlsanity.com/paper/1906.04009/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jun 2019.

## Built on by (most cited first, 18 of 18)

- [Prioritizing Samples in Reinforcement Learning with Reducible Loss](https://mlsanity.com/paper/2208.10483/) (2022)
- [Large Batch Experience Replay](https://mlsanity.com/paper/2110.01528/) (2021)
- [Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experience](https://mlsanity.com/paper/2109.11767/) (2021)
- [Revisiting Prioritized Experience Replay: A Value Perspective](https://mlsanity.com/paper/2102.03261/) (2021)
- [Convergence Results For Q-Learning With Experience Replay](https://mlsanity.com/paper/2112.04213/) (2021)
- [Soft Actor-Critic with Cross-Entropy Policy Optimization](https://mlsanity.com/paper/2112.11115/) (2021)
- [Adaptive Replay Buffer for Offline-to-Online Reinforcement Learning](https://mlsanity.com/paper/2512.10510/) (2025)
- [Explaining Off-Policy Actor-Critic From A Bias-Variance Perspective](https://mlsanity.com/paper/2110.02421/) (2021)
- [Model-Augmented Q-learning](https://mlsanity.com/paper/2102.03866/) (2021)
- [DearFSAC: An Approach to Optimizing Unreliable Federated Learning via Deep Reinforcement Learning](https://mlsanity.com/paper/2201.12701/) (2022)
- [Autonomous Navigation of Tractor-Trailer Vehicles through Roundabout Intersections](https://mlsanity.com/paper/2401.04980/) (2024)
- [COVR:Collaborative Optimization of VLMs and RL Agent for Visual-Based Control](https://mlsanity.com/paper/2601.06122/) (2026)
- [Replay For Safety](https://mlsanity.com/paper/2112.04229/) (2021)
- [Actor Loss of Soft Actor Critic Explained](https://mlsanity.com/paper/2112.15568/) (2021)
- [Deep Reinforcement Learning-Assisted Federated Learning for Robust Short-term Utility Demand Forecasting in Electricity Wholesale Markets](https://mlsanity.com/paper/2206.11715/) (2022)
- [When Does Non-Uniform Replay Matter in Reinforcement Learning?](https://mlsanity.com/paper/2605.10236/) (2026)
- [Continual-RL for Generalization in Autonomous Racing on the RoboRacer Platform](https://mlsanity.com/paper/2607.24320/) (2026)
- [Reward Inflation: A Healthy Stimulus for Reinforcement Learning](https://mlsanity.com/paper/2610.02545/) (2026)

## Abstract (arXiv)

Soft Actor-Critic (SAC) is an off-policy actor-critic deep reinforcement learning (DRL) algorithm based on maximum entropy reinforcement learning. By combining off-policy updates with an actor-critic formulation, SAC achieves state-of-the-art performance on a range of continuous-action benchmark tasks, outperforming prior on-policy and off-policy methods. The off-policy method employed by SAC samples data uniformly from past experience when performing parameter updates. We propose Emphasizing Recent Experience (ERE), a simple but powerful off-policy sampling technique, which emphasizes recently observed data while not forgetting the past. The ERE algorithm samples more aggressively from recent experience, and also orders the updates to ensure that updates from old data do not overwrite updates from new data. We compare vanilla SAC and SAC+ERE, and show that ERE is more sample efficient than vanilla SAC for continuous-action Mujoco tasks. We also consider combining SAC with Priority Experience Replay (PER), a scheme originally proposed for deep Q-learning which prioritizes the data based on temporal-difference (TD) error. We show that SAC+PER can marginally improve the sample efficiency performance of SAC, but much less so than SAC+ERE. Finally, we propose an algorithm which integrates ERE and PER and show that this hybrid algorithm can give the best results for some of the Mujoco tasks.