# Reward-Adaptive Reinforcement Learning: Dynamic Policy Gradient Optimization for Bipedal Locomotion

arXiv: https://arxiv.org/abs/2107.01908 · HTML: https://mlsanity.com/paper/2107.01908/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jul 2021.

## Built on by (most cited first, 13 of 13)

- [Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI](https://mlsanity.com/paper/2407.06886/) (2024)
- [BeamDojo: Learning Agile Humanoid Locomotion on Sparse Footholds](https://mlsanity.com/paper/2502.10363/) (2025)
- [Safe Reinforcement Learning with Dual Robustness](https://mlsanity.com/paper/2309.06835/) (2023)
- [Video2Reward: Generating Reward Function from Videos for Legged Robot Behavior Learning](https://mlsanity.com/paper/2412.05515/) (2024)
- [Deep Reinforcement Learning for Bipedal Locomotion: A Brief Survey](https://mlsanity.com/paper/2404.17070/) (2024)
- [Infer and Adapt: Bipedal Locomotion Reward Learning from Demonstrations via Inverse Reinforcement Learning](https://mlsanity.com/paper/2309.16074/) (2023)
- [FlowCritic: Bridging Value Estimation with Flow Matching in Reinforcement Learning](https://mlsanity.com/paper/2510.22686/) (2025)
- [Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Review](https://mlsanity.com/paper/2411.10268/) (2024)
- [Automated Hybrid Reward Scheduling via Large Language Models for Robotic Skill Learning](https://mlsanity.com/paper/2505.02483/) (2025)
- [Shared Growth of Graph Neural Networks via Prompted Free-direction Knowledge Distillation](https://mlsanity.com/paper/2307.00534/) (2023)
- [Bridging the Gap between Newton-Raphson Method and Regularized Policy Iteration](https://mlsanity.com/paper/2310.07211/) (2023)
- [UACER: An Uncertainty-Adaptive Critic Ensemble Framework for Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/2512.10492/) (2025)
- [ReCal: Reward Calibration for RL-based LLM Routing](https://mlsanity.com/paper/2606.12479/) (2026)

## Abstract (arXiv)

Controlling a non-statically bipedal robot is challenging due to the complex dynamics and multi-criterion optimization involved. Recent works have demonstrated the effectiveness of deep reinforcement learning (DRL) for simulation and physical robots. In these methods, the rewards from different criteria are normally summed to learn a single value function. However, this may cause the loss of dependency information between hybrid rewards and lead to a sub-optimal policy. In this work, we propose a novel reward-adaptive reinforcement learning for biped locomotion, allowing the control policy to be simultaneously optimized by multiple criteria using a dynamic mechanism. The proposed method applies a multi-head critic to learn a separate value function for each reward component. This leads to hybrid policy gradient. We further propose dynamic weight, allowing each component to optimize the policy with different priorities. This hybrid and dynamic policy gradient (HDPG) design makes the agent learn more efficiently. We show that the proposed method outperforms summed-up-reward approaches and is able to transfer to physical robots. The sim-to-real and MuJoCo results further demonstrate the effectiveness and generalization of HDPG.