mlsanityData to 7 Oct 2026

Deep Reinforcement Learning from Hierarchical Preference Design

arXiv 2309.02632 ↗First posted 6 Sep 2023cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Reward design is a fundamental, yet challenging aspect of reinforcement learning (RL). Researchers typically utilize feedback signals from the environment to handcraft a reward function, but this process is not always effective due to the varying scale and intricate dependencies of the feedback signals. This paper shows by exploiting certain structures, one can ease the reward design process. Specifically, we propose a hierarchical reward modeling framework -- HERON for scenarios: (I) The feedback signals naturally present hierarchy; (II) The reward is sparse, but with less important surrogate feedback to help policy learning. Both scenarios allow us to design a hierarchical decision tree induced by the importance ranking of the feedback signals to compare RL trajectories. With such preference data, we can then train a reward model for policy learning. We apply HERON to several RL applications, and we find that our framework can not only train high performing agents on a variety of difficult tasks, but also provide additional benefits such as improved sample efficiency and robustness. Our code is available at urlhttps://github.com/abukharin3/HERON.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 27 of its 51 references that are in the map. First posted on arXiv 6 Sep 2023.

Builds on

27 of 51 references.

References · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  10. 2015Trust Region Policy Optimizationcited by 2,503
  11. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  12. 2021Measuring Coding Challenge Competence With APPScited by 879
15 more
  1. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  2. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  3. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  4. 2020Learning to Utilize Shaping Rewards: A New Approach of Reward Shapingcited by 112
  5. 2023Execution-based Code Generation using Deep Reinforcement Learningcited by 95
  6. 2020Deep Reinforcement Learning for Intelligent Transportation Systems: A Surveycited by 94
  7. 2020FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Financecited by 86
  8. 2016Multi-Objective Deep Reinforcement Learningcited by 81
  9. 2018Learning Navigation Behaviors End-to-End with AutoRLcited by 70
  10. 2020Active Preference-Based Gaussian Process Regression for Reward Learningcited by 70
  11. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  12. 2019Evolving Rewards to Automate Reinforcement Learningcited by 28
  13. 2020Learning Dense Rewards for Contact-Rich Manipulation Taskscited by 18
  14. 2019Integrating independent and centralized multi-agent reinforcement learning for traffic signal network optimizationcited by 12
  15. 2022Automated Reinforcement Learning: An Overviewcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Shaping Sparse Rewards in Reinforcement Learning: A Semi-supervised Approachcited by 2
  2. 2026Balancing Multiple Objectives in Urban Traffic Control with Reinforcement Learning from AI Feedbackcited by 1
  3. 2026RHyVE: Competence-Aware Verification and Phase-Aware Deployment for LLM-Generated Reward Hypothesescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024PIPER: Primitive-Informed Preference-based Hierarchical Reinforcement Learning via Hindsight Relabeling
  2. 2025Reward Models in Deep Reinforcement Learning: A Survey
  3. 2024Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach
  4. 2018Hierarchical Imitation and Reinforcement Learning
  5. 2026RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction
  6. 2025Which Rewards Matter? Reward Selection for Reinforcement Learning under Limited Feedback
  7. 2023Is RLHF More Difficult than Standard RL?
  8. 2024Binary Reward Labeling: Bridging Offline Preference and Reward-Based Reinforcement Learning