Sufficiency of Zeroth-Order Reward Shaping for Policy Gradient in Stabilization Control
On the map: Deep reinforcement learning methods → Deep RL algorithms and representations
Abstract (arXiv)
Reward shaping is fundamental to modern robotic control with deep reinforcement learning (RL), yet practitioners still rely heavily on heuristic principles borrowed from classical optimal control and trajectory optimization. Existing methods rarely distinguish reward terms that are intrinsic to the control objective from numerical regularizers, leading to brittle hyperparameter tuning. To determine which quantities a reward must contain, we study the stabilization control problem with a focus on zeroth-order (configuration) and first-order (velocity) information. We theoretically and empirically demonstrate that policy gradient methods can successfully solve stabilization tasks without first-order reward terms, adding such terms can instead introduce severe sensitivity as their scale grows. Conversely, our findings confirm that reward functions must be zeroth-order complete over goal-relevant coordinates, while the first-order state remains necessary in the policy observation under our low-dissipation assumptions. Overall, these results provide actionable and principled guidance for reward design in robotic RL.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 21 of its 52 references that are in the map. First posted on arXiv 28 Sep 2026.
Builds on
21 of 52 references.
Closest ancestors · 1
Same direction, newest first
Other references · 20
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2017Hindsight Experience Replaycited by 1,136
- 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
- 2017Deep Reinforcement Learning that Matterscited by 754
- 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 604
- 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
- 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 512
- 2019Learning agile and dynamic motor skills for legged robotscited by 504
- 2023Orbit: A Unified Simulation Framework for Interactive Robot Learning Environmentscited by 295
- 2023Reward Design with Language Modelscited by 261
- 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 188
- 2023Real-World Humanoid Locomotion with Reinforcement Learningcited by 187
8 more
- 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 167
- 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 165
- 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 151
- 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 134
- 2016Learning values across many orders of magnitudecited by 85
- 2024DrEureka: Language Model Guided Sim-To-Real Transfercited by 83
- 2022Deep Black-Box Reinforcement Learning with Movement Primitivescited by 20
- 2025Reinforcement Learning with Segment Feedbackcited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradients
- 2024Combining Automated Optimisation of Hyperparameters and Reward Shape
- 2026Decoupling Task and Behavior: A Two-Stage Reward Curriculum in Reinforcement Learning for Robotics
- 2024Dense Dynamics-Aware Reward Synthesis: Integrating Prior Experience with Demonstrations
- 2024Stage-Wise Reward Shaping for Acrobatic Robots: A Constrained Multi-Objective Reinforcement Learning Approach
- 2024Revisiting Sparse Rewards for Goal-Reaching Reinforcement Learning
- 2024Curriculum Reinforcement Learning for Complex Reward Functions
- 2020Learning of Long-Horizon Sparse-Reward Robotic Manipulator Tasks with Base Controllers