mlsanityData to 7 Oct 2026

Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learning

arXiv 2408.03029 ↗First posted 6 Aug 2024cs.LG cs.AIcited by 21

Not on the map. Why?

Abstract (arXiv)

Reward shaping is a technique in reinforcement learning that addresses the sparse-reward problem by providing more frequent and informative rewards. We introduce a self-adaptive and highly efficient reward shaping mechanism that incorporates success rates derived from historical experiences as shaped rewards. The success rates are sampled from Beta distributions, which dynamically evolve from uncertain to reliable values as data accumulates. Initially, the shaped rewards exhibit more randomness to encourage exploration, while over time, the increasing certainty enhances exploitation, naturally balancing exploration and exploitation. Our approach employs Kernel Density Estimation (KDE) combined with Random Fourier Features (RFF) to derive the Beta distributions, providing a computationally efficient, non-parametric, and learning-free solution for high-dimensional continuous state spaces. Our method is validated on various tasks with extremely sparse rewards, demonstrating notable improvements in sample efficiency and convergence stability over relevant baselines.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 48 of its 82 references that are in the map. First posted on arXiv 6 Aug 2024.

Builds on

48 of 82 references.

References · 48

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2016Generative Adversarial Imitation Learningcited by 1,576
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  6. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  7. 2018Exploration by Random Network Distillationcited by 966
  8. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  9. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  10. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  11. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  12. 2016Cooperative Inverse Reinforcement Learningcited by 298
36 more
  1. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
  2. 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
  3. 2017Noisy Networks for Explorationcited by 262
  4. 2017Count-Based Exploration with Neural Density Modelscited by 255
  5. 2017One-Shot Imitation Learningcited by 232
  6. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  7. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 158
  8. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  9. 2018Count-Based Exploration with the Successor Representationcited by 112
  10. 2020Learning to Utilize Shaping Rewards: A New Approach of Reward Shapingcited by 112
  11. 2018On Learning Intrinsic Rewards for Policy Gradient Methodscited by 100
  12. 2019Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewardscited by 76
  13. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  14. 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
  15. 2018Diversity-Driven Exploration Strategy for Deep Reinforcement Learningcited by 65
  16. 2022Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexitycited by 59
  17. 2023Controllability-Aware Unsupervised Skill Discoverycited by 49
  18. 2017Count-Based Exploration in Feature Space for Reinforcement Learningcited by 48
  19. 2017EX2: Exploration with Exemplar Models for Deep Reinforcement Learningcited by 42
  20. 2021Heuristic-Guided Reinforcement Learningcited by 41
  21. 2021Self-Supervised Online Reward Shaping in Sparse-Reward Environmentscited by 34
  22. 2024Exploration and Anti-Exploration with Distributional Random Network Distillationcited by 32
  23. 2023Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learningcited by 28
  24. 2018DORA The Explorer: Directed Outreaching Reinforcement Action-Selectioncited by 27
  25. 2021How to Stay Curious while Avoiding Noisy TVs using Aleatoric Uncertainty Estimationcited by 27
  26. 2023Behavior Alignment via Reward Function Optimizationcited by 16
  27. 2020Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Modelscited by 14
  28. 2024SEABO: A Simple Search-Based Method for Offline Imitation Learningcited by 14
  29. 2024Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learningcited by 13
  30. 2021Learning to Shape Rewards using a Game of Two Partnerscited by 9
  31. 2022Utilizing Prior Solutions for Reward Shaping and Composition in Entropy-Regularized Reinforcement Learningcited by 8
  32. 2023How to Fine-tune the Model: Unified Model Shift and Model Bias Policy Optimizationcited by 6
  33. 2024Random Latent Exploration for Deep Reinforcement Learningcited by 6
  34. 2024Uncertainty-Aware Reward-Free Exploration with General Function Approximationcited by 5
  35. 2021Risk Averse Bayesian Reward Learning for Autonomous Navigation from Human Demonstrationcited by 4
  36. 2021Inverse Optimal Control Adapted to the Noise Characteristics of the Human Sensorimotor Systemcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 12; 2026: 8.

21 papers.

Sort:

By area:

  1. 2025TempSamp-R1: Effective Temporal Sampling with Reinforcement Fine-Tuning for Video LLMscited by 15
  2. 2025Skill Expansion and Composition in Parameter Spacecited by 13
  3. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  4. 2026A Subgoal-driven Framework for Improving Long-Horizon LLM Agentscited by 10
  5. 2025Adaptive Multi-prompt Contrastive Network for Few-shot Out-of-distribution Detectioncited by 8
  6. 2025Bootstrapped Reward Shapingcited by 3
  7. 2026The Art of Efficient Reasoning: Data, Reward, and Optimizationcited by 3
  8. 2025TW-CRL: Time-Weighted Contrastive Reward Learning for Efficient Inverse Reinforcement Learningcited by 2
  9. 2025Exploration by Random Distribution Distillationcited by 1
  10. 2025Exploration by Random Reward Perturbationcited by 1
  11. 2025Topology Enhanced MARL for Multi-Agent Cooperative Decision-Making of CAVscited by 1
  12. 2026Occupancy Reward Shaping: Improving Credit Assignment for Offline Goal-Conditioned Reinforcement Learningcited by 1
9 more
  1. 2026Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMscited by 1
  2. 2024Centralized Reward Agent for Knowledge Sharing and Transfer in Multi-Task Reinforcement Learningcited by 0
  3. 2025Causal Policy Learning in Reinforcement Learning: Backdoor-Adjusted Soft Actor-Criticcited by 0
  4. 2025DRTA: Dynamic Reward Scaling for Reinforcement Learning in Time Series Anomaly Detectioncited by 0
  5. 2025Learning Upper Lower Value Envelopes to Shape Online RL: A Principled Approachcited by 0
  6. 2026Quality-Aware Exploration Budget Allocation for Cooperative Multi-Agent Reinforcement Learningcited by 0
  7. 2026Learning Process Rewards via Success Visitation Matching for Efficient RLcited by 0
  8. 2026Information-Based Exploration via Random Features for Reinforcement Learningcited by 0
  9. 2026Meta-Learned Reward Shaping for Reinforcement Learning from Human Feedbackcited by 0