# Not Only Rewards But Also Constraints: Applications on Legged Robot Locomotion

arXiv: https://arxiv.org/abs/2308.12517 · HTML: https://mlsanity.com/paper/2308.12517/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 74 references that are in the map.
- First posted on arXiv 24 Aug 2023.

## Builds on (most cited first, 30 of 48)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning](https://mlsanity.com/paper/2109.11978/) (2021)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Reward Constrained Policy Optimization](https://mlsanity.com/paper/1805.11074/) (2018)
- [Learning robust perceptive locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2201.08117/) (2022)
- [Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates](https://mlsanity.com/paper/1610.00633/) (2016)
- [Risk-Constrained Reinforcement Learning with Percentile Risk Criteria](https://mlsanity.com/paper/1512.01629/) (2015)
- [Deep Dynamics Models for Learning Dexterous Manipulation](https://mlsanity.com/paper/1909.11652/) (2019)
- [Projection-Based Constrained Policy Optimization](https://mlsanity.com/paper/2010.03152/) (2020)
- [Real-World Humanoid Locomotion with Reinforcement Learning](https://mlsanity.com/paper/2303.03381/) (2023)
- [Rapid Locomotion via Reinforcement Learning](https://mlsanity.com/paper/2205.02824/) (2022)
- [Emergence of Locomotion Behaviours in Rich Environments](https://mlsanity.com/paper/1707.02286/) (2017)
- [ANYmal Parkour: Learning Agile Navigation for Quadrupedal Robots](https://mlsanity.com/paper/2306.14874/) (2023)
- [Legged Locomotion in Challenging Terrains using Egocentric Vision](https://mlsanity.com/paper/2211.07638/) (2022)
- [CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guarantee](https://mlsanity.com/paper/2011.05869/) (2020)
- [IPO: Interior-point Policy Optimization under Constraints](https://mlsanity.com/paper/1910.09615/) (2019)
- [Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotion](https://mlsanity.com/paper/2210.10044/) (2022)
- [Conservative Safety Critics for Exploration](https://mlsanity.com/paper/2010.14497/) (2020)
- [BADGR: An Autonomous Self-Supervised Learning-Based Navigation System](https://mlsanity.com/paper/2002.05700/) (2020)

## Built on by (most cited first, 30 of 47)

- [DrEureka: Language Model Guided Sim-To-Real Transfer](https://mlsanity.com/paper/2406.01967/) (2024)
- [One Policy to Run Them All: an End-to-end Learning Approach to Multi-Embodiment Locomotion](https://mlsanity.com/paper/2409.06366/) (2024)
- [CaT: Constraints as Terminations for Legged Locomotion Reinforcement Learning](https://mlsanity.com/paper/2403.18765/) (2024)
- [LocoFormer: Generalist Locomotion via Long-context Adaptation](https://mlsanity.com/paper/2509.23745/) (2025)
- [SoloParkour: Constrained Reinforcement Learning for Visual Locomotion from Privileged Experience](https://mlsanity.com/paper/2409.13678/) (2024)
- [Stage-Wise Reward Shaping for Acrobatic Robots: A Constrained Multi-Objective Reinforcement Learning Approach](https://mlsanity.com/paper/2409.15755/) (2024)
- [Safety Filters for Black-Box Dynamical Systems by Learning Discriminating Hyperplanes](https://mlsanity.com/paper/2402.05279/) (2024)
- [A Learning Framework for Diverse Legged Robot Locomotion Using Barrier-Based Style Rewards](https://mlsanity.com/paper/2409.15780/) (2024)
- [VLANeXt: Recipes for Building Strong VLA Models](https://mlsanity.com/paper/2602.18532/) (2026)
- [Gradient Shaping for Multi-Constraint Safe Reinforcement Learning](https://mlsanity.com/paper/2312.15127/) (2023)
- [One Filter to Deploy Them All: Robust Safety for Quadrupedal Navigation in Unknown Environments](https://mlsanity.com/paper/2412.09989/) (2024)
- [PA-LOCO: Learning Perturbation-Adaptive Locomotion for Quadruped Robots](https://mlsanity.com/paper/2407.04224/) (2024)
- [SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer](https://mlsanity.com/paper/2509.18648/) (2025)
- [DecAP: Decaying Action Priors for Accelerated Imitation Learning of Torque-Based Legged Locomotion Policies](https://mlsanity.com/paper/2310.05714/) (2023)
- [Automatic Environment Shaping is the Next Frontier in RL](https://mlsanity.com/paper/2407.16186/) (2024)
- [RAPID: Robust and Agile Planner Using Inverse Reinforcement Learning for Vision-Based Drone Navigation](https://mlsanity.com/paper/2502.02054/) (2025)
- [Learning Semantic Traversability with Egocentric Video and Automated Annotation Strategy](https://mlsanity.com/paper/2406.02989/) (2024)
- [TAR: Teacher-Aligned Representations via Contrastive Learning for Quadrupedal Locomotion](https://mlsanity.com/paper/2503.20839/) (2025)
- [Legged Robot State Estimation With Invariant Extended Kalman Filter Using Neural Measurement Network](https://mlsanity.com/paper/2402.00366/) (2024)
- [AnyBipe: An End-to-End Framework for Training and Deploying Bipedal Robots Guided by Large Language Models](https://mlsanity.com/paper/2409.08904/) (2024)
- [Spectral Normalization for Lipschitz-Constrained Policies on Learning Humanoid Locomotion](https://mlsanity.com/paper/2504.08246/) (2025)
- [Reinforcement Learning from Wild Animal Videos](https://mlsanity.com/paper/2412.04273/) (2024)
- [QuietPaw: Learning Quadrupedal Locomotion with Versatile Noise Preference Alignment](https://mlsanity.com/paper/2503.05035/) (2025)
- [Toward Reliable Sim-to-Real Predictability for MoE-based Robust Quadrupedal Locomotion](https://mlsanity.com/paper/2602.00678/) (2026)
- [Learning Rapid Turning, Aerial Reorientation, and Balancing using Manipulator as a Tail](https://mlsanity.com/paper/2407.10420/) (2024)
- [BarlowWalk: Self-supervised Representation Learning for Legged Robot Terrain-adaptive Locomotion](https://mlsanity.com/paper/2508.00939/) (2025)
- [Constrained Reinforcement Learning for Unstable Point-Feet Bipedal Locomotion Applied to the Bolt Robot](https://mlsanity.com/paper/2508.02194/) (2025)
- [MASH: Cooperative-Heterogeneous Multi-Agent Reinforcement Learning for Single Humanoid Robot Locomotion](https://mlsanity.com/paper/2508.10423/) (2025)
- [Periodic Skill Discovery](https://mlsanity.com/paper/2511.03187/) (2025)
- [ECO: Energy-Constrained Optimization with Reinforcement Learning for Humanoid Walking](https://mlsanity.com/paper/2602.06445/) (2026)

## Abstract (arXiv)

Several earlier studies have shown impressive control performance in complex robotic systems by designing the controller using a neural network and training it with model-free reinforcement learning. However, these outstanding controllers with natural motion style and high task performance are developed through extensive reward engineering, which is a highly laborious and time-consuming process of designing numerous reward terms and determining suitable reward coefficients. In this work, we propose a novel reinforcement learning framework for training neural network controllers for complex robotic systems consisting of both rewards and constraints. To let the engineers appropriately reflect their intent to constraints and handle them with minimal computation overhead, two constraint types and an efficient policy optimization algorithm are suggested. The learning framework is applied to train locomotion controllers for several legged robots with different morphology and physical attributes to traverse challenging terrains. Extensive simulation and real-world experiments demonstrate that performant controllers can be trained with significantly less reward engineering, by tuning only a single reward coefficient. Furthermore, a more straightforward and intuitive engineering process can be utilized, thanks to the interpretability and generalizability of constraints. The summary video is available at https://youtu.be/KAlm3yskhvM.