mlsanityData to 7 Oct 2026

Self-Supervised Online Reward Shaping in Sparse-Reward Environments

arXiv 2103.04529 ↗First posted 8 Mar 2021cs.LG cs.ROcited by 34

Not on the map. Why?

Abstract (arXiv)

We introduce Self-supervised Online Reward Shaping (SORS), which aims to improve the sample efficiency of any RL algorithm in sparse-reward environments by automatically densifying rewards. The proposed framework alternates between classification-based reward inference and policy update steps -- the original sparse reward provides a self-supervisory signal for reward inference by ranking trajectories that the agent observes, while the policy update is performed with the newly inferred, typically dense reward function. We introduce theory that shows that, under certain conditions, this alteration of the reward function will not change the optimal policy of the original MDP, while potentially increasing learning speed significantly. Experimental results on several sparse-reward environments demonstrate that, across multiple domains, the proposed algorithm is not only significantly more sample efficient than a standard RL baseline using sparse rewards, but, at times, also achieves similar sample efficiency compared to when hand-designed dense reward functions are used.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 19 of its 38 references that are in the map. First posted on arXiv 8 Mar 2021.

Builds on

19 of 38 references.

References · 19

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  6. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  7. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  8. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  9. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  10. 2017Inverse Reward Designcited by 246
  11. 2020Fast Differentiable Sorting and Rankingcited by 196
  12. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 164
7 more
  1. 2019Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewardscited by 76
  2. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  3. 2021Modular Deep Reinforcement Learning for Continuous Motion Planning with Temporal Logiccited by 38
  4. 2019Reward Shaping via Meta-Learningcited by 30
  5. 2018Machine Teaching for Inverse Reinforcement Learning: Algorithms and Applicationscited by 28
  6. 2018Generative Adversarial Self-Imitation Learningcited by 25
  7. 2020Active Task-Inference-Guided Deep Inverse Reinforcement Learningcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 0; 2023: 8; 2024: 9; 2025: 8; 2026: 7.

34 papers.

Sort:

By area:

  1. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learningcited by 55
  2. 2023Safe Reinforcement Learning via Probabilistic Logic Shieldscited by 41
  3. 2025Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewardscited by 29
  4. 2024Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learningcited by 21
  5. 2021Offline Preference-Based Apprenticeship Learningcited by 17
  6. 2024Offline Model-Based Optimization by Learning to Rankcited by 13
  7. 2023Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysiscited by 10
  8. 2025Multi-Stage Manipulation with Demonstration-Augmented Reward, Policy, and World Model Learningcited by 10
  9. 2024DrS: Learning Reusable Dense Rewards for Multi-Stage Taskscited by 9
  10. 2023Towards Safe Autonomous Driving Policies using a Neuro-Symbolic Deep Reinforcement Learning Approachcited by 7
  11. 2025Preference-based Multi-Objective Reinforcement Learningcited by 7
  12. 2025Attention-Based Reward Shaping for Sparse and Delayed Rewardscited by 5
22 more
  1. 2023Intrinsically motivated graph exploration using network theories of human curiositycited by 3
  2. 2024Informativeness of Reward Functions in Reinforcement Learningcited by 3
  3. 2024Enhancing Inverse Reinforcement Learning through Encoding Dynamic Information in Reward Shapingcited by 3
  4. 2021On the Benefits of Inducing Local Lipschitzness for Robust Generative Adversarial Imitation Learningcited by 2
  5. 2024ETGL-DDPG: A Deep Deterministic Policy Gradient Algorithm for Sparse Reward Continuous Controlcited by 2
  6. 2025Shaping Sparse Rewards in Reinforcement Learning: A Semi-supervised Approachcited by 2
  7. 2025Beyond expected value: geometric mean optimization for long-term policy performance in reinforcement learningcited by 2
  8. 2026E2HiL: Entropy-Guided Sample Selection for Efficient Real-World Human-in-the-Loop Reinforcement Learningcited by 2
  9. 2024On the Sample Efficiency of Abstractions and Potential-Based Reward Shaping in Reinforcement Learningcited by 1
  10. 2024Appraisal-Guided Proximal Policy Optimization: Modeling Psychological Disorders in Dynamic Grid Worldcited by 1
  11. 2026Automating Potential-based Reward Shaping with Vision Language Model Guidancecited by 1
  12. 2023A reinforcement learning path planning approach for range-only underwater target localization with autonomous vehiclescited by 0
  13. 2023Embedding Contextual Information through Reward Shaping in Multi-Agent Learning: A Case Study from Google Footballcited by 0
  14. 2023Mining of Single-Class by Active Learning for Semantic Segmentationcited by 0
  15. 2024Centralized Reward Agent for Knowledge Sharing and Transfer in Multi-Task Reinforcement Learningcited by 0
  16. 2025Finite-time Convergence Analysis of Actor-Critic with Evolving Rewardcited by 0
  17. 2025Learning to Compress Graphs via Dual Agents for Consistent Topological Robustness Evaluationcited by 0
  18. 2026STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Ordercited by 0
  19. 2026PRISM: Parallel Reward Integration with Symmetry for MORLcited by 0
  20. 2026ARMS: Automatic Reward Shaping for Sparse-Reward Multi-Agent Reinforcement Learningcited by 0
  21. 2026Learning Process Rewards via Success Visitation Matching for Efficient RLcited by 0
  22. 2026iADD: Improving Alignment and Diversity in Diffusion Policy Optimizationcited by 0