mlsanityData to 14 Oct 2026

CaT: Constraints as Terminations for Legged Locomotion Reinforcement Learning

arXiv 2403.18765 ↗First posted 27 Mar 2024cs.RO cs.LGcited by 23

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation

Abstract (arXiv)

Deep Reinforcement Learning (RL) has demonstrated impressive results in solving complex robotic tasks such as quadruped locomotion. Yet, current solvers fail to produce efficient policies respecting hard constraints. In this work, we advocate for integrating constraints into robot learning and present Constraints as Terminations (CaT), a novel constrained RL algorithm. Departing from classical constrained RL formulations, we reformulate constraints through stochastic terminations during policy learning: any violation of a constraint triggers a probability of terminating potential future rewards the RL agent could attain. We propose an algorithmic approach to this formulation, by minimally modifying widely used off-the-shelf RL algorithms in robot learning (such as Proximal Policy Optimization). Our approach leads to excellent constraint adherence without introducing undue complexity and computational overhead, thus mitigating barriers to broader adoption. Through empirical evaluation on the real quadruped robot Solo crossing challenging obstacles, we demonstrate that CaT provides a compelling solution for incorporating constraints into RL frameworks. Videos and code are available at https://constraints-as-terminations.github.io.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 40 of its 58 references that are in the map. First posted on arXiv 27 Mar 2024.

Builds on

40 of 58 references.

References · 40

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  4. 2017Constrained Policy Optimizationcited by 983
  5. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  6. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 512
  7. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 508
  8. 2017Safe Reinforcement Learning via Shieldingcited by 449
  9. 2018Reward Constrained Policy Optimizationcited by 417
  10. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  11. 2022DayDreamer: World Models for Physical Robot Learningcited by 400
  12. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
28 more
  1. 2018Sim-to-Real: Learning Agile Locomotion For Quadruped Robotscited by 319
  2. 2021Curriculum Learning: A Surveycited by 259
  3. 2020Learning Agile Robotic Locomotion Skills by Imitating Animalscited by 251
  4. 2022Rapid Locomotion via Reinforcement Learningcited by 168
  5. 2023Extreme Parkour with Legged Robotscited by 164
  6. 2017OSQP: An Operator Splitting Solver for Quadratic Programscited by 161
  7. 2023ANYmal Parkour: Learning Agile Navigation for Quadrupedal Robotscited by 152
  8. 2022Legged Locomotion in Challenging Terrains using Egocentric Visioncited by 151
  9. 2023Robot Parkour Learningcited by 149
  10. 2019IPO: Interior-point Policy Optimization under Constraintscited by 132
  11. 2022Adversarial Motion Priors Make Good Substitutes for Complex Reward Functionscited by 89
  12. 2022A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learningcited by 83
  13. 2021Minimizing Energy Consumption Leads to the Emergence of Gaits in Legged Robotscited by 76
  14. 2024Agile But Safe: Learning Collision-Free High-Speed Legged Locomotioncited by 73
  15. 2022Penalized Proximal Policy Optimization for Safe Reinforcement Learningcited by 68
  16. 2020Dynamics Randomization Revisited:A Case Study for Quadrupedal Locomotioncited by 59
  17. 2023Not Only Rewards But Also Constraints: Applications on Legged Robot Locomotioncited by 47
  18. 2023Learning Vision-Based Bipedal Locomotion for Challenging Terraincited by 42
  19. 2022Safe Reinforcement Learning for Legged Locomotioncited by 35
  20. 2022CPG-RL: Learning Central Pattern Generators for Quadruped Locomotioncited by 34
  21. 2022Learning Torque Control for Quadrupedal Locomotioncited by 29
  22. 2019An Open Torque-Controlled Modular Robot Architecture for Legged Locomotion Researchcited by 22
  23. 2021Robust High-speed Running for Quadruped Robots via Deep Reinforcement Learningcited by 22
  24. 2023Controlling the Solo12 Quadruped Robot with Deep Reinforcement Learningcited by 16
  25. 2023Evaluation of Constrained Reinforcement Learning Algorithms for Legged Locomotioncited by 13
  26. 2016An Efficient Optimal Planning and Control Framework For Quadrupedal Locomotioncited by 12
  27. 2023Learning Terrain-Adaptive Locomotion with Agile Behaviors by Imitating Animalscited by 10
  28. 2019SL1M: Sparse L1-norm Minimization for contact planning on uneven terraincited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 6; 2026: 14.

23 papers.

Sort:

By area:

  1. 2025RSL-RL: A Learning Library for Robotics Researchcited by 49
  2. 2024SoloParkour: Constrained Reinforcement Learning for Visual Locomotion from Privileged Experiencecited by 14
  3. 2025Dynamic object goal pushing with mobile manipulators through model-free constrained reinforcement learningcited by 5
  4. 2025First Order Model-Based RL through Decoupled Backpropagationcited by 5
  5. 2025Latent Conditioned Loco-Manipulation Using Motion Priorscited by 3
  6. 2024Reinforcement Learning from Wild Animal Videoscited by 2
  7. 2026CACTO-BIC: Scalable Actor-Critic Learning via Biased Sampling and GPU-Accelerated Trajectory Optimizationcited by 2
  8. 2026Flip Stunts on Bicycle Robots using Iterative Motion Imitationcited by 2
  9. 2026Direct Dynamic Retargeting for Humanoid Imitation Learning from Videoscited by 2
  10. 2024Learning Rapid Turning, Aerial Reorientation, and Balancing using Manipulator as a Tailcited by 1
  11. 2025Constrained Reinforcement Learning for Unstable Point-Feet Bipedal Locomotion Applied to the Bolt Robotcited by 1
  12. 2026Coupled Local and Global World Models for Efficient First Order RLcited by 1
11 more
  1. 2026Safe Execution of RL Policies Via Acceleration-Based CBF-QP Constraint Enforcement for Real-World Robotic Deploymentscited by 1
  2. 2025Whole-Body Constrained Learning for Legged Locomotion via Hierarchical Optimizationcited by 0
  3. 2026KiRAS: Keyframe Guided Self-Imitation for Robust and Adaptive Skill Learning in Quadruped Robotscited by 0
  4. 2026LoComposition: Terrain-Adaptive Energy-Efficient Quadruped Locomotion without Gait Priorscited by 0
  5. 2026Bicycle Acrobatics with Reinforcement Learningcited by 0
  6. 2026The Dually Flat Geometry of Planning as Inferencecited by 0
  7. 2026FALCON-S: Fixed-wing ground-effect Aerodynamics Simulator and Flight Control Learning Suitecited by 0
  8. 2026SCoCaT: Success Conditioned Constrained Reinforcement Learning for Spacecraft Dockingcited by 0
  9. 2026Smoothness as a Constraint for Stable Humanoid Locomotioncited by 0
  10. 2026DODGER: Safety-Guided Reinforcement Learning for Robot Navigation Among Dynamic Obstaclescited by 0
  11. 2026Phase-Gated Constrained Reinforcement Learning for End-to-End Capture and Detumbling of Non-Cooperative Targetscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023A Real-World Quadrupedal Locomotion Benchmark for Offline Reinforcement Learning
  2. 2026Learning Gait-Aware Quadruped Locomotion with Temporal Logic Specifications
  3. 2025Gait in Eight: Efficient On-Robot Learning for Omnidirectional Quadruped Locomotion
  4. 2025Training Directional Locomotion for Quadrupedal Low-Cost Robotic Systems via Deep Reinforcement Learning
  5. 2024Leveraging Symmetry in RL-based Legged Locomotion Control
  6. 2024A Learning Framework for Diverse Legged Robot Locomotion Using Barrier-Based Style Rewards
  7. 2020Learning to Walk in the Real World with Minimal Human Effort
  8. 2024Constrained Skill Discovery: Quadruped Locomotion with Unsupervised Reinforcement Learning