mlsanityData to 7 Oct 2026

Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexity

arXiv 2210.09579 ↗First posted 18 Oct 2022cs.LG cs.AIcited by 59

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning provides an automated framework for learning behaviors from high-level reward specifications, but in practice the choice of reward function can be crucial for good results -- while in principle the reward only needs to specify what the task is, in reality practitioners often need to design more detailed rewards that provide the agent with some hints about how the task should be completed. The idea of this type of ``reward-shaping'' has been often discussed in the literature, and is often a critical part of practical applications, but there is relatively little formal characterization of how the choice of reward shaping can yield benefits in sample complexity. In this work, we build on the framework of novelty-based exploration to provide a simple scheme for incorporating shaped rewards into RL along with an analysis tool to show that particular choices of reward shaping provably improve sample efficiency. We characterize the class of problems where these gains are expected to be significant and show how this can be connected to practical algorithms in the literature. We confirm that these results hold in practice in an experimental evaluation, providing an insight into the mechanisms through which reward shaping can significantly improve the complexity of reinforcement learning while retaining asymptotic performance.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Builds on 42 of its 61 references that are in the map. First posted on arXiv 18 Oct 2022.

Builds on

42 of 61 references.

References · 42

  1. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  2. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  3. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  4. 2018Exploration by Random Network Distillationcited by 966
  5. 2018Learning Dexterous In-Hand Manipulationcited by 643
  6. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  7. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  8. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  9. 2018Is Q-learning Provably Efficient?cited by 485
  10. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  11. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  12. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
30 more
  1. 2019Provably Efficient Exploration in Policy Optimizationcited by 203
  2. 2019Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Boundcited by 199
  3. 2019Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Boundscited by 194
  4. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  5. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  6. 2020Almost Optimal Model-Free Reinforcement Learning via Reference-Advantage Decompositioncited by 127
  7. 2019Optimism in Reinforcement Learning with Generalized Linear Function Approximationcited by 111
  8. 2020Provably Efficient Reinforcement Learning for Discounted MDPs with Feature Mappingcited by 102
  9. 2019Frequentist Regret Bounds for Randomized Least-Squares Value Iterationcited by 100
  10. 2016Corralling a Band of Bandit Algorithmscited by 99
  11. 2020PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learningcited by 91
  12. 2020Towards Playing Full MOBA Games with Deep Reinforcement Learningcited by 78
  13. 2019Provably Efficient Q-Learning with Low Switching Costcited by 77
  14. 2020Model Selection in Contextual Stochastic Bandit Problemscited by 71
  15. 2018Efficient Bias-Span-Constrained Exploration-Exploitation in Reinforcement Learningcited by 65
  16. 2017PRM-RL: Long-range Robotic Navigation Tasks by Combining Reinforcement Learning and Sampling-based Planningcited by 64
  17. 2020Q-learning with Logarithmic Regretcited by 59
  18. 2017Hybrid Reward Architecture for Reinforcement Learningcited by 57
  19. 2021Breaking the Sample Complexity Barrier to Regret-Optimal Model-Free Reinforcement Learningcited by 51
  20. 2020Regret Bound Balancing and Elimination for Model Selection in Bandits and RLcited by 45
  21. 2021UCB Momentum Q-learning: Correcting the bias without forgettingcited by 45
  22. 2021Heuristic-Guided Reinforcement Learningcited by 41
  23. 2019Tight Regret Bounds for Model-Based Reinforcement Learning with Greedy Policiescited by 38
  24. 2021Beyond Value-Function Gaps: Improved Instance-Dependent Regret Bounds for Episodic Reinforcement Learningcited by 33
  25. 2017Posterior sampling for reinforcement learning: worst-case regret boundscited by 28
  26. 2021MURAL: Meta-Learning Uncertainty-Aware Rewards for Outcome-Driven Reinforcement Learningcited by 24
  27. 2020Regret Balancing for Bandit and RL Model Selectioncited by 20
  28. 2021Computational Benefits of Intermediate Rewards for Goal-Reaching Policy Learningcited by 17
  29. 2021Can Q-Learning be Improved with Advice?cited by 13
  30. 2020Towards Tractable Optimism in Model-Based Reinforcement Learningcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 14; 2024: 19; 2025: 17; 2026: 7.

59 papers.

Sort:

By area:

  1. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  2. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  3. 2023General In-Hand Object Rotation with Vision and Touchcited by 87
  4. 2024Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applicationscited by 34
  5. 2024A Large Language Model-Driven Reward Design Framework via Dynamic Feedback for Reinforcement Learningcited by 29
  6. 2025Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewardscited by 29
  7. 2023RLIF: Interactive Imitation Learning as Reinforcement Learningcited by 24
  8. 2023Accelerating Exploration with Unlabeled Prior Datacited by 21
  9. 2024Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learningcited by 21
  10. 2024Real-World Offline Reinforcement Learning from Vision Language Model Feedbackcited by 20
  11. 2023Reward Shaping for Happier Autonomous Cyber Security Agentscited by 19
  12. 2022Understanding the Complexity Gains of Single-Task RL with a Curriculumcited by 16
38 more
  1. 2025Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuningcited by 15
  2. 2025TempSamp-R1: Effective Temporal Sampling with Reinforcement Fine-Tuning for Video LLMscited by 15
  3. 2022Learning Representations that Enable Generalization in Assistive Taskscited by 11
  4. 2023When is Agnostic Reinforcement Learning Statistically Tractable?cited by 8
  5. 2025ReDit: Reward Dithering for Improved LLM Policy Optimizationcited by 8
  6. 2024Assessing the Zero-Shot Capabilities of LLMs for Action Evaluation in RLcited by 7
  7. 2024Online Preference-based Reinforcement Learning with Self-augmented Feedback from Large Language Modelcited by 7
  8. 2024Automatic Environment Shaping is the Next Frontier in RLcited by 6
  9. 2025Automatic Reward Shaping from Confounded Offline Datacited by 6
  10. 2023Guaranteeing Control Requirements via Reward Shaping in Reinforcement Learningcited by 5
  11. 2024Combining Automated Optimisation of Hyperparameters and Reward Shapecited by 5
  12. 2024Benchmarking Smoothness and Reducing High-Frequency Oscillations in Continuous Control Policiescited by 5
  13. 2025ARM-FM: Automated Reward Machines via Foundation Models for Compositional Reinforcement Learningcited by 5
  14. 2023FoMo Rewards: Can we cast foundation models as reward functions?cited by 4
  15. 2024Advancing Investment Frontiers: Industry-grade Deep Reinforcement Learning for Portfolio Optimizationcited by 4
  16. 2025Reinforcement Learning with Physics-Informed Symbolic Program Priors for Zero-Shot Wireless Indoor Navigationcited by 4
  17. 2024Hindsight Preference Learning for Offline Preference-based Reinforcement Learningcited by 3
  18. 2024Enhancing Inverse Reinforcement Learning through Encoding Dynamic Information in Reward Shapingcited by 3
  19. 2025Bootstrapped Reward Shapingcited by 3
  20. 2025Reward Training Wheels: Adaptive Auxiliary Rewards for Robotics Reinforcement Learningcited by 3
  21. 2023Toward Computationally Efficient Inverse Reinforcement Learning via Reward Shapingcited by 2
  22. 2024Quasimetric Value Functions with Dense Rewardscited by 2
  23. 2025Learning Explainable Dense Reward Shapes via Bayesian Optimizationcited by 2
  24. 2025Mash, Spread, Slice! Learning to Manipulate Object States via Visual Spatial Progresscited by 2
  25. 2025PROF: An LLM-based Reward Code Preference Optimization Framework for Offline Imitation Learningcited by 2
  26. 2023Robust Knowledge Transfer in Tiered Reinforcement Learningcited by 1
  27. 2023Contextual Pre-planning on Reward Machine Abstractions for Enhanced Transfer in Deep Reinforcement Learningcited by 1
  28. 2023Tracking Control for a Spherical Pendulum via Curriculum Reinforcement Learningcited by 1
  29. 2023Quantifying Assistive Robustness Via the Natural-Adversarial Frontiercited by 1
  30. 2024On the Sample Efficiency of Abstractions and Potential-Based Reward Shaping in Reinforcement Learningcited by 1
  31. 2024Efficient Imitation Without Demonstrations via Value-Penalized Auxiliary Control from Examplescited by 1
  32. 2025An Automated Reinforcement Learning Reward Design Framework with Large Language Model for Cooperative Platoon Coordinationcited by 1
  33. 2026Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMscited by 1
  34. 2023Mining of Single-Class by Active Learning for Semantic Segmentationcited by 0
  35. 2024Sample Efficient Reinforcement Learning by Automatically Learning to Compose Subtaskscited by 0
  36. 2024Heuristic Algorithm-based Action Masking Reinforcement Learning (HAAM-RL) with Ensemble Inference Methodcited by 0
  37. 2024Extremum-Seeking Action Selection for Accelerating Policy Optimizationcited by 0
  38. 2025Reinforcement Learning on Reconfigurable Hardware: Overcoming Material Variability in Laser Material Processingcited by 0

50 most cited of 59 All 59 (JSON)