mlsanityData to 14 Oct 2026

Bridging the Gap: Enabling Soft Actor Critic for High Performance Legged Locomotion

arXiv 2605.24975 ↗First posted 24 May 2026cs.RO cs.AI cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation

Abstract (arXiv)

Proximal Policy Optimization (PPO) has become the de facto standard for training legged robots, thanks to its robustness and scalability in massively parallel simulation environments like IsaacLab. However, its on-policy nature makes it inherently sample-inefficient, preventing its use for continuous adaptation and fine-tuning on real hardware. Soft Actor-Critic (SAC), by contrast, is an off-policy algorithm that can reuse past experience, making it a natural candidate for sim-to-real transfer workflows where the same algorithm can be used both in simulation and for online learning on the real robot. Despite these advantages, SAC has consistently failed to match PPO's empirical performance in massively parallel training settings. This work identifies the root causes of this gap and introduces targeted modifications, covering policy initialization, timeout-aware critic targets, and multi-step return estimation, that enable SAC to train stably at scale. Evaluated across multiple legged robot platforms and diverse locomotion tasks, our approach closes the performance gap with PPO entirely.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 15 of its 27 references that are in the map. First posted on arXiv 24 May 2026.

Builds on

15 of 27 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026What Matters for Simulation to Online Reinforcement Learning on Real Robots23 Feb
  2. 2025Learning Sim-to-Real Humanoid Locomotion in 15 Minutes1 Dec
  3. 2025RSL-RL: A Learning Library for Robotics Research13 Sep
  4. 2025FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Control28 May

Other references · 11

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  3. 2018Exploration by Random Network Distillationcited by 993
  4. 2024Humanoid Parkour Learningcited by 96
  5. 2024Simplifying Deep Temporal Difference Learningcited by 78
  6. 2025MuJoCo Playgroundcited by 78
  7. 2021Legged Robots that Keep on Learning: Fine-Tuning Locomotion Policies in the Real Worldcited by 71
  8. 2025Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Roboticscited by 51
  9. 2024DextrAH-RGB: Visuomotor Policies to Grasp Anything with Dexterous Handscited by 45
  10. 2025Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot Learningcited by 24
  11. 2025Uncertainty-Aware Robotic World Model Makes Offline Model-Based Reinforcement Learning Work on Real Robotscited by 17

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026ContractionPPO: Certified Reinforcement Learning via Differentiable Contraction Layers
  2. 2026GPO: Growing Policy Optimization for Legged Robot Locomotion and Whole-Body Control
  3. 2026Towards Bridging the Gap between Large-Scale Pretraining and Efficient Finetuning for Humanoid Control
  4. 2025Unlocking the Potential of Soft Actor-Critic for Imitation Learning
  5. 2023DecAP: Decaying Action Priors for Accelerated Imitation Learning of Torque-Based Legged Locomotion Policies
  6. 2026A Geometric Approach to Soft Actor-Critic with Zonotopes for Locomotion Learning
  7. 2025Development of a PPO-Reinforcement Learned Walking Tripedal Soft-Legged Robot using SOFA
  8. 2025Pretraining in Actor-Critic Reinforcement Learning for Locomotion