# Torque-based Deep Reinforcement Learning for Task-and-Robot Agnostic Learning on Bipedal Robots Using Sim-to-Real Transfer

arXiv: https://arxiv.org/abs/2304.09434 · HTML: https://mlsanity.com/paper/2304.09434/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Apr 2023.

## Built on by (most cited first, 22 of 22)

- [Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion Control](https://mlsanity.com/paper/2401.16889/) (2024)
- [Whole-body Humanoid Robot Locomotion with Human Reference](https://mlsanity.com/paper/2402.18294/) (2024)
- [The Reality Gap in Robotics: Challenges, Solutions, and Best Practices](https://mlsanity.com/paper/2510.20808/) (2025)
- [FACET: Force-Adaptive Control via Impedance Reference Tracking for Legged Robots](https://mlsanity.com/paper/2505.06883/) (2025)
- [Tune to Learn: How Controller Gains Shape Robot Policy Learning](https://mlsanity.com/paper/2604.02523/) (2026)
- [DecAP: Decaying Action Priors for Accelerated Imitation Learning of Torque-Based Legged Locomotion Policies](https://mlsanity.com/paper/2310.05714/) (2023)
- [Spectral Normalization for Lipschitz-Constrained Policies on Learning Humanoid Locomotion](https://mlsanity.com/paper/2504.08246/) (2025)
- [Robust Humanoid Walking on Compliant and Uneven Terrain with Deep Reinforcement Learning](https://mlsanity.com/paper/2504.13619/) (2025)
- [Disturbance-Aware Adaptive Compensation in Hybrid Force-Position Locomotion Policy for Legged Robots](https://mlsanity.com/paper/2506.00472/) (2025)
- [LiPS: Large-Scale Humanoid Robot Reinforcement Learning with Parallel-Series Structures](https://mlsanity.com/paper/2503.08349/) (2025)
- [Efficiently Learning Robust Torque-based Locomotion Through Reinforcement with Model-Based Supervision](https://mlsanity.com/paper/2601.16109/) (2026)
- [Learning Bipedal Walking for Humanoid Robots in Challenging Environments with Obstacle Avoidance](https://mlsanity.com/paper/2410.08212/) (2024)
- [Variable Stiffness for Robust Locomotion through Reinforcement Learning](https://mlsanity.com/paper/2502.09436/) (2025)
- [SATA: Safe and Adaptive Torque-Based Locomotion Policies Inspired by Animal Learning](https://mlsanity.com/paper/2502.12674/) (2025)
- [GPO: Growing Policy Optimization for Legged Robot Locomotion and Whole-Body Control](https://mlsanity.com/paper/2601.20668/) (2026)
- [Benchmarking Action Spaces in Reinforcement Learning for Vision-based Robotic Manipulation](https://mlsanity.com/paper/2606.18594/) (2026)
- [Learning Generic and Dynamic Locomotion of Humanoids Across Discrete Terrains](https://mlsanity.com/paper/2405.17227/) (2024)
- [TOLEBI: Learning Fault-Tolerant Bipedal Locomotion via Online Status Estimation and Fallibility Rewards](https://mlsanity.com/paper/2602.05596/) (2026)
- [Sim-to-Real of Humanoid Locomotion Policies via Joint Torque Space Perturbation Injection](https://mlsanity.com/paper/2603.21853/) (2026)
- [Behavior Cloning Under PD Control: A Finite-Horizon Theory of Gain-Dependent Error Amplification](https://mlsanity.com/paper/2604.14484/) (2026)
- [Comparative Study on Agility, Efficiency, and Impact Absorption of Bipedal Robots with Active Toes](https://mlsanity.com/paper/2606.19699/) (2026)
- [Enhancing Sim2Real Transfer for Torque-Controlled Robots through Real2Sim Dynamics Estimation and Reinforcement Learning](https://mlsanity.com/paper/2608.22629/) (2026)

## Abstract (arXiv)

In this paper, we review the question of which action space is best suited for controlling a real biped robot in combination with Sim2Real training. Position control has been popular as it has been shown to be more sample efficient and intuitive to combine with other planning algorithms. However, for position control gain tuning is required to achieve the best possible policy performance. We show that instead, using a torque-based action space enables task-and-robot agnostic learning with less parameter tuning and mitigates the sim-to-reality gap by taking advantage of torque control's inherent compliance. Also, we accelerate the torque-based-policy training process by pre-training the policy to remain upright by compensating for gravity. The paper showcases the first successful sim-to-real transfer of a torque-based deep reinforcement learning policy on a real human-sized biped robot. The video is available at https://youtu.be/CR6pTS39VRE.