mlsanityData to 7 Oct 2026

Learning to Utilize Shaping Rewards: A New Approach of Reward Shaping

arXiv 2011.02669 ↗First posted 5 Nov 2020cs.LG cs.AIcited by 112

Not on the map. Why?

Abstract (arXiv)

Reward shaping is an effective technique for incorporating domain knowledge into reinforcement learning (RL). Existing approaches such as potential-based reward shaping normally make full use of a given shaping reward function. However, since the transformation of human knowledge into numeric reward values is often imperfect due to reasons such as human cognitive bias, completely utilizing the shaping reward function may fail to improve the performance of RL algorithms. In this paper, we consider the problem of adaptively utilizing a given shaping reward function. We formulate the utilization of shaping rewards as a bi-level optimization problem, where the lower level is to optimize policy using the shaping rewards and the upper level is to optimize a parameterized shaping weight function for true reward maximization. We formally derive the gradient of the expected true reward with respect to the shaping weight function parameters and accordingly propose three learning algorithms based on different assumptions. Experiments in sparse-reward cartpole and MuJoCo environments show that our algorithms can fully exploit beneficial shaping rewards, and meanwhile ignore unbeneficial shaping rewards or even transform them into beneficial ones.

Key facts. Cited by 112 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 12; 2023: 23; 2024: 21; 2025: 33; 2026: 13.

112 papers.

Sort:

By area:

  1. 2025Self-Adapting Language Modelscited by 71
  2. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  3. 2021Heuristic-Guided Reinforcement Learningcited by 41
  4. 2024Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHFcited by 41
  5. 2024Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applicationscited by 34
  6. 2021Core Challenges in Embodied Vision-Language Planningcited by 30
  7. 2022Guarantees for Epsilon-Greedy Reinforcement Learning with Function Approximationcited by 29
  8. 2025Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewardscited by 29
  9. 2021ReLAX: Reinforcement Learning Agent eXplainer for Arbitrary Predictive Modelscited by 25
  10. 2022Redeeming Intrinsic Rewards via Constrained Optimizationcited by 24
  11. 2024Latent Reward: LLM-Empowered Credit Assignment in Episodic Reinforcement Learningcited by 22
  12. 2024Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learningcited by 21
38 more
  1. 2023Interpretable Reward Redistribution in Reinforcement Learning: A Causal Approachcited by 17
  2. 2023State2Explanation: Concept-Based Explanations to Benefit Agent Learning and User Understandingcited by 17
  3. 2023Behavior Alignment via Reward Function Optimizationcited by 16
  4. 2023Mind the Gap: Offline Policy Optimization for Imperfect Rewardscited by 15
  5. 2025Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchronycited by 13
  6. 2023Principal-Agent Reward Shaping in MDPscited by 12
  7. 2021A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learningcited by 11
  8. 2025Towards Safe Reasoning in Large Reasoning Models via Corrective Interventioncited by 11
  9. 2025Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimizationcited by 11
  10. 2024Bilevel reinforcement learning via the development of hyper-gradient without lower-level convexitycited by 10
  11. 2025From Screens to Scenes: A Survey of Embodied AI in Healthcarecited by 10
  12. 2025On The Sample Complexity Bounds In Bilevel Reinforcement Learningcited by 10
  13. 2021Learning to Shape Rewards using a Game of Two Partnerscited by 9
  14. 2022Generalization in Deep Reinforcement Learning for Robotic Navigation by Reward Shapingcited by 8
  15. 2024Principal-Agent Reinforcement Learning: Orchestrating AI Agents with Contractscited by 8
  16. 2022Learning Synthetic Environments and Reward Networks for Reinforcement Learningcited by 6
  17. 2024Automatic Environment Shaping is the Next Frontier in RLcited by 6
  18. 2021Discovering and Exploiting Sparse Rewards in a Learned Behavior Spacecited by 5
  19. 2022Advice Conformance Verification by Reinforcement Learning agents for Human-in-the-Loopcited by 5
  20. 2023Guaranteeing Control Requirements via Reward Shaping in Reinforcement Learningcited by 5
  21. 2023Towards Skilled Population Curriculum for Multi-Agent Reinforcement Learningcited by 4
  22. 2023FoMo Rewards: Can we cast foundation models as reward functions?cited by 4
  23. 2025LearningFlow: Automated Policy Learning Workflow for Urban Driving with Large Language Modelscited by 4
  24. 2025Stabilizing Long-term Multi-turn Reinforcement Learning with Gated Rewardscited by 4
  25. 2025Tree-Guided Diffusion Plannercited by 4
  26. 2025Fine-tuning Flow Matching Generative Models with Intermediate Feedbackcited by 4
  27. 2025Sharpness-Guided Group Relative Policy Optimization via Probability Shapingcited by 4
  28. 2021Goal-constrained Sparse Reinforcement Learning for End-to-End Drivingcited by 3
  29. 2021Model-Based Safe Reinforcement Learning with Time-Varying State and Control Constraints: An Application to Intelligent Vehiclescited by 3
  30. 2022Impact Makes a Sound and Sound Makes an Impact: Sound Guides Representations and Explorationscited by 3
  31. 2022Safe Reinforcement Learning with Contrastive Risk Predictioncited by 3
  32. 2023Sample-efficient Real-time Planning with Curiosity Cross-Entropy Method and Contrastive Learningcited by 3
  33. 2023Deep Reinforcement Learning from Hierarchical Preference Designcited by 3
  34. 2023Learning Extrinsic Dexterity with Parameterized Manipulation Primitivescited by 3
  35. 2024Transformable Gaussian Reward Function for Socially-Aware Navigation with Deep Reinforcement Learningcited by 3
  36. 2024BCR-DRL: Behavior- and Context-aware Reward for Deep Reinforcement Learning in Human-AI Coordinationcited by 3
  37. 2024Enhancing Inverse Reinforcement Learning through Encoding Dynamic Information in Reward Shapingcited by 3
  38. 2024Efficient Model-Based Reinforcement Learning Through Optimistic Thompson Samplingcited by 3

50 most cited of 112 All 112 (JSON)