# Unlocking the Potential of Soft Actor-Critic for Imitation Learning

arXiv: https://arxiv.org/abs/2509.24539 · HTML: https://mlsanity.com/paper/2509.24539/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)
- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 14 of 14)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Learning Agile Robotic Locomotion Skills by Imitating Animals](https://mlsanity.com/paper/2004.00784/) (2020)
- [GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning](https://mlsanity.com/paper/1810.05762/) (2018)
- [Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions](https://mlsanity.com/paper/2203.15103/) (2022)
- [Lifelike Agility and Play in Quadrupedal Robots using Reinforcement Learning and Generative Pre-trained Models](https://mlsanity.com/paper/2308.15143/) (2023)
- [Versatile Skill Control via Self-supervised Adversarial Imitation of Unlabeled Mixed Motions](https://mlsanity.com/paper/2209.07899/) (2022)
- [Imitation Learning from MPC for Quadrupedal Multi-Gait Control](https://mlsanity.com/paper/2103.14331/) (2021)
- [Learning Terrain-Adaptive Locomotion with Agile Behaviors by Imitating Animals](https://mlsanity.com/paper/2308.03273/) (2023)
- [Imitating and Finetuning Model Predictive Control for Robust and Symmetric Quadrupedal Locomotion](https://mlsanity.com/paper/2311.02304/) (2023)
- [Learning Highly Dynamic Behaviors for Quadrupedal Robots](https://mlsanity.com/paper/2402.13473/) (2024)
- [Integrating Diffusion-based Multi-task Learning with Online Reinforcement Learning for Robust Quadruped Robot Control](https://mlsanity.com/paper/2507.05674/) (2025)
- [Tiny Reinforcement Learning for Quadruped Locomotion using Decision Transformers](https://mlsanity.com/paper/2402.13201/) (2024)
- [In-between Motion Generation Based Multi-Style Quadruped Robot Locomotion](https://mlsanity.com/paper/2507.23053/) (2025)

## Abstract (arXiv)

Learning-based methods have enabled robots to acquire bio-inspired movements with increasing levels of naturalness and adaptability. Among these, Imitation Learning (IL) has proven effective in transferring complex motion patterns from animals to robotic systems. However, current state-of-the-art frameworks predominantly rely on Proximal Policy Optimization (PPO), an on-policy algorithm that prioritizes stability over sample efficiency and policy generalization. This paper proposes a novel IL framework that combines Adversarial Motion Priors (AMP) with the off-policy Soft Actor-Critic (SAC) algorithm to overcome these limitations. This integration leverages replay-driven learning and entropy-regularized exploration, enabling naturalistic behavior and task execution, improving data efficiency and robustness. We evaluate the proposed approach (AMP+SAC) on quadruped gaits involving multiple reference motions and diverse terrains. Experimental results demonstrate that the proposed framework not only maintains stable task execution but also achieves higher imitation rewards compared to the widely used AMP+PPO method. These findings highlight the potential of an off-policy IL formulation for advancing motion generation in robotics.