# CaT: Constraints as Terminations for Legged Locomotion Reinforcement Learning

arXiv: https://arxiv.org/abs/2403.18765 · HTML: https://mlsanity.com/paper/2403.18765/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 40 of its 58 references that are in the map.
- First posted on arXiv 27 Mar 2024.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 30 of 40)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning](https://mlsanity.com/paper/2109.11978/) (2021)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [Reward Constrained Policy Optimization](https://mlsanity.com/paper/1805.11074/) (2018)
- [Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation](https://mlsanity.com/paper/2106.13281/) (2021)
- [DayDreamer: World Models for Physical Robot Learning](https://mlsanity.com/paper/2206.14176/) (2022)
- [Risk-Constrained Reinforcement Learning with Percentile Risk Criteria](https://mlsanity.com/paper/1512.01629/) (2015)
- [Sim-to-Real: Learning Agile Locomotion For Quadruped Robots](https://mlsanity.com/paper/1804.10332/) (2018)
- [Curriculum Learning: A Survey](https://mlsanity.com/paper/2101.10382/) (2021)
- [Learning Agile Robotic Locomotion Skills by Imitating Animals](https://mlsanity.com/paper/2004.00784/) (2020)
- [Rapid Locomotion via Reinforcement Learning](https://mlsanity.com/paper/2205.02824/) (2022)
- [Extreme Parkour with Legged Robots](https://mlsanity.com/paper/2309.14341/) (2023)
- [OSQP: An Operator Splitting Solver for Quadratic Programs](https://mlsanity.com/paper/1711.08013/) (2017)
- [ANYmal Parkour: Learning Agile Navigation for Quadrupedal Robots](https://mlsanity.com/paper/2306.14874/) (2023)
- [Legged Locomotion in Challenging Terrains using Egocentric Vision](https://mlsanity.com/paper/2211.07638/) (2022)
- [Robot Parkour Learning](https://mlsanity.com/paper/2309.05665/) (2023)
- [IPO: Interior-point Policy Optimization under Constraints](https://mlsanity.com/paper/1910.09615/) (2019)
- [Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions](https://mlsanity.com/paper/2203.15103/) (2022)
- [A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learning](https://mlsanity.com/paper/2208.07860/) (2022)
- [Minimizing Energy Consumption Leads to the Emergence of Gaits in Legged Robots](https://mlsanity.com/paper/2111.01674/) (2021)
- [Agile But Safe: Learning Collision-Free High-Speed Legged Locomotion](https://mlsanity.com/paper/2401.17583/) (2024)
- [Penalized Proximal Policy Optimization for Safe Reinforcement Learning](https://mlsanity.com/paper/2205.11814/) (2022)
- [Dynamics Randomization Revisited:A Case Study for Quadrupedal Locomotion](https://mlsanity.com/paper/2011.02404/) (2020)
- [Not Only Rewards But Also Constraints: Applications on Legged Robot Locomotion](https://mlsanity.com/paper/2308.12517/) (2023)
- [Learning Vision-Based Bipedal Locomotion for Challenging Terrain](https://mlsanity.com/paper/2309.14594/) (2023)

## Built on by (most cited first, 23 of 23)

- [RSL-RL: A Learning Library for Robotics Research](https://mlsanity.com/paper/2509.10771/) (2025)
- [SoloParkour: Constrained Reinforcement Learning for Visual Locomotion from Privileged Experience](https://mlsanity.com/paper/2409.13678/) (2024)
- [Dynamic object goal pushing with mobile manipulators through model-free constrained reinforcement learning](https://mlsanity.com/paper/2502.01546/) (2025)
- [First Order Model-Based RL through Decoupled Backpropagation](https://mlsanity.com/paper/2509.00215/) (2025)
- [Latent Conditioned Loco-Manipulation Using Motion Priors](https://mlsanity.com/paper/2509.16061/) (2025)
- [Reinforcement Learning from Wild Animal Videos](https://mlsanity.com/paper/2412.04273/) (2024)
- [CACTO-BIC: Scalable Actor-Critic Learning via Biased Sampling and GPU-Accelerated Trajectory Optimization](https://mlsanity.com/paper/2602.19699/) (2026)
- [Flip Stunts on Bicycle Robots using Iterative Motion Imitation](https://mlsanity.com/paper/2603.27944/) (2026)
- [Direct Dynamic Retargeting for Humanoid Imitation Learning from Videos](https://mlsanity.com/paper/2605.23762/) (2026)
- [Learning Rapid Turning, Aerial Reorientation, and Balancing using Manipulator as a Tail](https://mlsanity.com/paper/2407.10420/) (2024)
- [Constrained Reinforcement Learning for Unstable Point-Feet Bipedal Locomotion Applied to the Bolt Robot](https://mlsanity.com/paper/2508.02194/) (2025)
- [Coupled Local and Global World Models for Efficient First Order RL](https://mlsanity.com/paper/2602.06219/) (2026)
- [Safe Execution of RL Policies Via Acceleration-Based CBF-QP Constraint Enforcement for Real-World Robotic Deployments](https://mlsanity.com/paper/2607.14488/) (2026)
- [Whole-Body Constrained Learning for Legged Locomotion via Hierarchical Optimization](https://mlsanity.com/paper/2506.05115/) (2025)
- [KiRAS: Keyframe Guided Self-Imitation for Robust and Adaptive Skill Learning in Quadruped Robots](https://mlsanity.com/paper/2603.15179/) (2026)
- [LoComposition: Terrain-Adaptive Energy-Efficient Quadruped Locomotion without Gait Priors](https://mlsanity.com/paper/2606.15896/) (2026)
- [Bicycle Acrobatics with Reinforcement Learning](https://mlsanity.com/paper/2608.00880/) (2026)
- [The Dually Flat Geometry of Planning as Inference](https://mlsanity.com/paper/2609.04005/) (2026)
- [FALCON-S: Fixed-wing ground-effect Aerodynamics Simulator and Flight Control Learning Suite](https://mlsanity.com/paper/2609.06046/) (2026)
- [SCoCaT: Success Conditioned Constrained Reinforcement Learning for Spacecraft Docking](https://mlsanity.com/paper/2609.06061/) (2026)
- [Smoothness as a Constraint for Stable Humanoid Locomotion](https://mlsanity.com/paper/2609.24552/) (2026)
- [DODGER: Safety-Guided Reinforcement Learning for Robot Navigation Among Dynamic Obstacles](https://mlsanity.com/paper/2609.38873/) (2026)
- [Phase-Gated Constrained Reinforcement Learning for End-to-End Capture and Detumbling of Non-Cooperative Targets](https://mlsanity.com/paper/2610.15491/) (2026)

## Abstract (arXiv)

Deep Reinforcement Learning (RL) has demonstrated impressive results in solving complex robotic tasks such as quadruped locomotion. Yet, current solvers fail to produce efficient policies respecting hard constraints. In this work, we advocate for integrating constraints into robot learning and present Constraints as Terminations (CaT), a novel constrained RL algorithm. Departing from classical constrained RL formulations, we reformulate constraints through stochastic terminations during policy learning: any violation of a constraint triggers a probability of terminating potential future rewards the RL agent could attain. We propose an algorithmic approach to this formulation, by minimally modifying widely used off-the-shelf RL algorithms in robot learning (such as Proximal Policy Optimization). Our approach leads to excellent constraint adherence without introducing undue complexity and computational overhead, thus mitigating barriers to broader adoption. Through empirical evaluation on the real quadruped robot Solo crossing challenging obstacles, we demonstrate that CaT provides a compelling solution for incorporating constraints into RL frameworks. Videos and code are available at https://constraints-as-terminations.github.io.