mlsanityData to 14 Oct 2026

Dense Dynamics-Aware Reward Synthesis: Integrating Prior Experience with Demonstrations

arXiv 2412.01114 ↗First posted 2 Dec 2024cs.LGcited by 2

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Many continuous control problems can be formulated as sparse-reward reinforcement learning (RL) tasks. In principle, online RL methods can automatically explore the state space to solve each new task. However, discovering sequences of actions that lead to a non-zero reward becomes exponentially more difficult as the task horizon increases. Manually shaping rewards can accelerate learning for a fixed task, but it is an arduous process that must be repeated for each new environment. We introduce a systematic reward-shaping framework that distills the information contained in 1) a task-agnostic prior data set and 2) a small number of task-specific expert demonstrations, and then uses these priors to synthesize dense dynamics-aware rewards for the given task. This supervision substantially accelerates learning in our experiments, and we provide analysis demonstrating how the approach can effectively guide online learning agents to faraway goals.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 30 of its 39 references that are in the map. First posted on arXiv 2 Dec 2024.

Builds on

30 of 39 references.

References · 30

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  4. 2016Generative Adversarial Imitation Learningcited by 1,597
  5. 2017Hindsight Experience Replaycited by 1,136
  6. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  7. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  8. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  9. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 511
  10. 2018Behavioral Cloning from Observationcited by 492
  11. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  12. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
18 more
  1. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  2. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 194
  3. 2017DART: Noise Injection for Robust Imitation Learningcited by 191
  4. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 154
  5. 2019Planning with Goal-Conditioned Policiescited by 149
  6. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 134
  7. 2020Primal Wasserstein Imitation Learningcited by 121
  8. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gapcited by 81
  9. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  10. 2021Heuristic-Guided Reinforcement Learningcited by 42
  11. 2023Provable Guarantees for Generative Behavior Cloning: Bridging Low-Level Stability and High-Level Behaviorcited by 33
  12. 2023CCIL: Continuity-based Data Augmentation for Corrective Imitation Learningcited by 29
  13. 2022Lyapunov Design for Robust and Efficient Robotic Reinforcement Learningcited by 28
  14. 2023Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shapingcited by 25
  15. 2022Back to the Manifold: Recovering from Out-of-Distribution Statescited by 20
  16. 2022Monte Carlo Augmented Actor-Critic for Sparse Reward Deep Reinforcement Learning from Suboptimal Demonstrationscited by 14
  17. 2021A Pragmatic Look at Deep Imitation Learningcited by 9
  18. 2023Learning Sparse Control Tasks from Pixels by Latent Nearest-Neighbor-Guided Explorationscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2026Emergent Dexterity via Diverse Resets and Large-Scale Reinforcement Learningcited by 10
  2. 2025Average-Reward Soft Actor-Criticcited by 6

Similar, not cited

Close in wording, no citation either way

  1. 2025DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning
  2. 2026A Unified Framework for Dynamic Reward Shaping in Reinforcement Learning
  3. 2026ARMS: Automatic Reward Shaping for Sparse-Reward Multi-Agent Reinforcement Learning
  4. 2024Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learning
  5. 2025Attention-Based Reward Shaping for Sparse and Delayed Rewards
  6. 2024Revisiting Sparse Rewards for Goal-Reaching Reinforcement Learning
  7. 2025Fill in the Blanks: Accelerating Q-Learning with a Handful of Demonstrations in Sparse Reward Settings
  8. 2024Adaptive Reward Design for Reinforcement Learning