Learning Without Time-Based Embodiment Resets in Soft-Actor Critic
Not on the map. Why?
Abstract (arXiv)
When creating new reinforcement learning tasks, practitioners often accelerate the learning process by incorporating into the task several accessory components, such as breaking the environment interaction into independent episodes and frequently resetting the environment. Although they can enable the learning of complex intelligent behaviors, such task accessories can result in unnatural task setups and hinder long-term performance in the real world. In this work, we explore the challenges of learning without episode terminations and robot embodiment resets using the Soft Actor-Critic (SAC) algorithm. To learn without terminations, we present a continuing version of the SAC algorithm and show that, with simple modifications to the reward functions of existing tasks, continuing SAC can perform as well as or better than episodic SAC while reducing the sensitivity of performance to the value of the discount rate γ. On a modified Gym Reacher task, we investigate possible explanations for the failure of continuing SAC when learning without embodiment resets. Our results suggest that embodiment resets help with exploration of the state space in the SAC algorithm, and removing embodiment resets can lead to poor exploration of the state space and failure of or significantly slower learning. Finally, on additional simulated tasks and a real-robot vision task, we show that increasing the entropy of the policy when performance trends worse or remains static is an effective intervention for recovering the performance lost due to not using embodiment resets.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 28 of its 34 references that are in the map. First posted on arXiv 6 Dec 2025.
Builds on
28 of 34 references.
References · 28
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2016Layer Normalizationcited by 5,579
- 2015Trust Region Policy Optimizationcited by 2,503
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
- 2016OpenAI Gymcited by 1,684
- 2023Mastering Diverse Domains through World Modelscited by 1,193
- 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
- 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
- 2018Exploration by Random Network Distillationcited by 966
- 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
- 2020dm_control: Software and Tasks for Continuous Controlcited by 368
16 more
- 2021Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Interventioncited by 90
- 2017Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learningcited by 86
- 2017Time Limits in Reinforcement Learningcited by 72
- 2021Autonomous Reinforcement Learning: Formalism and Benchmarkingcited by 33
- 2024Exploration and Anti-Exploration with Distributional Random Network Distillationcited by 32
- 2021On-Policy Deep Reinforcement Learning for the Average-Reward Criterioncited by 29
- 2021Is High Variance Unavoidable in RL? A Case Study in Continuous Controlcited by 26
- 2021Average-Reward Reinforcement Learning with Trust Region Methodscited by 18
- 2024Reward Centeringcited by 16
- 2022Real-Time Reinforcement Learning for Vision-Based Robotics Utilizing Local and Remote Computerscited by 13
- 2024Revisiting Sparse Rewards for Goal-Reaching Reinforcement Learningcited by 12
- 2022Automating Reinforcement Learning with Example-based Resetscited by 11
- 2024RVI-SAC: Average Reward Off-Policy Deep Reinforcement Learningcited by 5
- 2024Intelligent Switching for Reset-Free RLcited by 4
- 2024Reset-free Reinforcement Learning with World Modelscited by 3
- 2025An Empirical Study of Deep Reinforcement Learning in Continuing Taskscited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.