# Evaluation of Constrained Reinforcement Learning Algorithms for Legged Locomotion

arXiv: https://arxiv.org/abs/2309.15430 · HTML: https://mlsanity.com/paper/2309.15430/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Sep 2023.

## Built on by (most cited first, 13 of 13)

- [One Policy to Run Them All: an End-to-end Learning Approach to Multi-Embodiment Locomotion](https://mlsanity.com/paper/2409.06366/) (2024)
- [CaT: Constraints as Terminations for Legged Locomotion Reinforcement Learning](https://mlsanity.com/paper/2403.18765/) (2024)
- [Learning coordinated badminton skills for legged manipulators](https://mlsanity.com/paper/2505.22974/) (2025)
- [SoloParkour: Constrained Reinforcement Learning for Visual Locomotion from Privileged Experience](https://mlsanity.com/paper/2409.13678/) (2024)
- [Stage-Wise Reward Shaping for Acrobatic Robots: A Constrained Multi-Objective Reinforcement Learning Approach](https://mlsanity.com/paper/2409.15755/) (2024)
- [A Learning Framework for Diverse Legged Robot Locomotion Using Barrier-Based Style Rewards](https://mlsanity.com/paper/2409.15780/) (2024)
- [Learning Whole-Body Loco-Manipulation for Omni-Directional Task Space Pose Tracking with a Wheeled-Quadrupedal-Manipulator](https://mlsanity.com/paper/2412.03012/) (2024)
- [SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer](https://mlsanity.com/paper/2509.18648/) (2025)
- [Reinforcement Learning from Wild Animal Videos](https://mlsanity.com/paper/2412.04273/) (2024)
- [Learning Rapid Turning, Aerial Reorientation, and Balancing using Manipulator as a Tail](https://mlsanity.com/paper/2407.10420/) (2024)
- [Whole-Body Constrained Learning for Legged Locomotion via Hierarchical Optimization](https://mlsanity.com/paper/2506.05115/) (2025)
- [Guiding Energy-Efficient Locomotion through Impact Mitigation Rewards](https://mlsanity.com/paper/2510.09543/) (2025)
- [Navigating the Proximity-Safety Balance: Constraint Decomposition for Human Following in Pedestrian Crowds](https://mlsanity.com/paper/2608.10056/) (2026)

## Abstract (arXiv)

Shifting from traditional control strategies to Deep Reinforcement Learning (RL) for legged robots poses inherent challenges, especially when addressing real-world physical constraints during training. While high-fidelity simulations provide significant benefits, they often bypass these essential physical limitations. In this paper, we experiment with the Constrained Markov Decision Process (CMDP) framework instead of the conventional unconstrained RL for robotic applications. We perform a comparative study of different constrained policy optimization algorithms to identify suitable methods for practical implementation. Our robot experiments demonstrate the critical role of incorporating physical constraints, yielding successful sim-to-real transfers, and reducing operational errors on physical systems. The CMDP formulation streamlines the training process by separately handling constraints from rewards. Our findings underscore the potential of constrained RL for the effective development and deployment of learned controllers in robotics.