# Stage-Wise Reward Shaping for Acrobatic Robots: A Constrained Multi-Objective Reinforcement Learning Approach

arXiv: https://arxiv.org/abs/2409.15755 · HTML: https://mlsanity.com/paper/2409.15755/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 24 Sep 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 24 of 24)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [Responsive Safety in Reinforcement Learning by PID Lagrangian Methods](https://mlsanity.com/paper/2007.03964/) (2020)
- [A Practical Guide to Multi-Objective Reinforcement Learning and Planning](https://mlsanity.com/paper/2103.09568/) (2021)
- [A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation](https://mlsanity.com/paper/1908.08342/) (2019)
- [Extreme Parkour with Legged Robots](https://mlsanity.com/paper/2309.14341/) (2023)
- [DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills](https://mlsanity.com/paper/1804.02717/) (2018)
- [Legged Locomotion in Challenging Terrains using Egocentric Vision](https://mlsanity.com/paper/2211.07638/) (2022)
- [Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behavior](https://mlsanity.com/paper/2212.03238/) (2022)
- [Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learning](https://mlsanity.com/paper/2105.08328/) (2021)
- [Concurrent Training of a Control Policy and a State Estimator for Dynamic and Robust Legged Locomotion](https://mlsanity.com/paper/2202.05481/) (2022)
- [Legged Robots that Keep on Learning: Fine-Tuning Locomotion Policies in the Real World](https://mlsanity.com/paper/2110.05457/) (2021)
- [Penalized Proximal Policy Optimization for Safe Reinforcement Learning](https://mlsanity.com/paper/2205.11814/) (2022)
- [PD-MORL: Preference-Driven Multi-Objective Reinforcement Learning Algorithm](https://mlsanity.com/paper/2208.07914/) (2022)
- [A Distributional View on Multi-Objective Policy Optimization](https://mlsanity.com/paper/2005.07513/) (2020)
- [Not Only Rewards But Also Constraints: Applications on Legged Robot Locomotion](https://mlsanity.com/paper/2308.12517/) (2023)
- [Barkour: Benchmarking Animal-level Agility with Quadruped Robots](https://mlsanity.com/paper/2305.14654/) (2023)
- [Robust and Versatile Bipedal Jumping Control through Reinforcement Learning](https://mlsanity.com/paper/2302.09450/) (2023)
- [Learning Agile Locomotion on Risky Terrains](https://mlsanity.com/paper/2311.10484/) (2023)
- [Evaluation of Constrained Reinforcement Learning Algorithms for Legged Locomotion](https://mlsanity.com/paper/2309.15430/) (2023)

## Built on by (most cited first, 13 of 13)

- [Learning Humanoid Standing-up Control across Diverse Postures](https://mlsanity.com/paper/2502.08378/) (2025)
- [STARE-VLA: Progressive Stage-Aware Reinforcement for Fine-Tuning Vision-Language-Action Models](https://mlsanity.com/paper/2512.05107/) (2025)
- [Gait in Eight: Efficient On-Robot Learning for Omnidirectional Quadruped Locomotion](https://mlsanity.com/paper/2503.08375/) (2025)
- [MARG: MAstering Risky Gap Terrains for Legged Robots with Elevation Mapping](https://mlsanity.com/paper/2509.20036/) (2025)
- [LAPP: Large Language Model Feedback for Preference-Driven Reinforcement Learning](https://mlsanity.com/paper/2504.15472/) (2025)
- [Bridge the Gap: Enhancing Quadruped Locomotion with Vertical Ground Perturbations](https://mlsanity.com/paper/2510.13488/) (2025)
- [Parental Guidance: Efficient Lifelong Learning through Evolutionary Distillation](https://mlsanity.com/paper/2503.18531/) (2025)
- [Learning Impact-Rich Rotational Maneuvers via Centroidal Velocity Rewards and Sim-to-Real Techniques: A One-Leg Hopper Flip Case Study](https://mlsanity.com/paper/2505.12222/) (2025)
- [Co-jump: Cooperative Jumping with Quadrupedal Robots via Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2602.10514/) (2026)
- [Reward-Zero: Language Embedding Driven Implicit Reward Mechanisms for Reinforcement Learning](https://mlsanity.com/paper/2603.09331/) (2026)
- [Learning Task-Invariant Properties via Dreamer: Enabling Efficient Policy Transfer for Quadruped Robots](https://mlsanity.com/paper/2604.02911/) (2026)
- [Robust Fall Recovery for Armless Bipedal-Wheeled Robots Via Force-Guided Learning](https://mlsanity.com/paper/2606.14270/) (2026)
- [Where Success Breaks: Failure-Boundary Learning for Robust Vision-Language-Action Models](https://mlsanity.com/paper/2609.06114/) (2026)

## Abstract (arXiv)

As the complexity of tasks addressed through reinforcement learning (RL) increases, the definition of reward functions also has become highly complicated. We introduce an RL method aimed at simplifying the reward-shaping process through intuitive strategies. Initially, instead of a single reward function composed of various terms, we define multiple reward and cost functions within a constrained multi-objective RL (CMORL) framework. For tasks involving sequential complex movements, we segment the task into distinct stages and define multiple rewards and costs for each stage. Finally, we introduce a practical CMORL algorithm that maximizes objectives based on these rewards while satisfying constraints defined by the costs. The proposed method has been successfully demonstrated across a variety of acrobatic tasks in both simulation and real-world environments. Additionally, it has been shown to successfully perform tasks compared to existing RL and constrained RL algorithms. Our code is available at https://github.com/rllab-snu/Stage-Wise-CMORL.