mlsanityData to 14 Oct 2026

Which Rewards Matter? Reward Selection for Reinforcement Learning under Limited Feedback

arXiv 2510.00144 ↗First posted 30 Sep 2025cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

The ability of reinforcement learning algorithms to learn effective policies is determined by the rewards available during training. However, for practical problems, obtaining large quantities of reward labels is often infeasible due to computational or financial constraints, particularly when relying on human feedback. When reinforcement learning must proceed with limited feedback -- only a fraction of samples get rewards labeled -- a fundamental question arises: which samples should be labeled to maximize policy performance? We formalize this problem of reward selection for reinforcement learning from limited feedback (RLLF), introducing a new problem formulation that facilitates the study of strategies for selecting impactful rewards. Two types of selection strategies are investigated: (i) heuristics that rely on reward-free information such as state visitation and partial value functions, and (ii) strategies pre-trained using auxiliary evaluative feedback. We find that critical subsets of rewards are those that (1) guide the agent along optimal trajectories, and (2) support recovery toward near-optimal behavior after deviations. Effective selection methods yield near-optimal policies with significantly fewer reward labels than full supervision, establishing reward selection as a powerful paradigm for scaling reinforcement learning in feedback-limited settings.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 49 references that are in the map. First posted on arXiv 30 Sep 2025.

Builds on

27 of 49 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  2. 2021Information Directed Reward Learning for Reinforcement Learning24 Feb
  3. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 24

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  4. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  5. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  6. 2016Automatic chemical design using a data-driven continuous representation of moleculescited by 765
  7. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  8. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 574
  9. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  10. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
  11. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 217
  12. 2017Deep Reinforcement Learning for De-Novo Drug Designcited by 161
12 more
  1. 2019Asking Easy Questions: A User-Friendly Approach to Active Reward Learningcited by 68
  2. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learningcited by 62
  3. 2020Interpretable Off-Policy Evaluation in Reinforcement Learning by Highlighting Influential Transitionscited by 41
  4. 2021Active Offline Policy Selectioncited by 26
  5. 2020Active Preference Learning using Maximum Regretcited by 21
  6. 2023Accelerating Exploration with Unlabeled Prior Datacited by 21
  7. 2020Active Reinforcement Learning: Observing Rewards at a Costcited by 20
  8. 2023Unsupervised Behavior Extraction via Random Intent Priorscited by 13
  9. 2024Leveraging Skills from Unlabeled Prior Data for Efficient Online Explorationcited by 13
  10. 2018Active Reinforcement Learning with Monte-Carlo Tree Searchcited by 11
  11. 2024Monitored Markov Decision Processescited by 10
  12. 2024Beyond Optimism: Exploration With Partially Observable Rewardscited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Reward Models in Deep Reinforcement Learning: A Survey
  2. 2023On The Fragility of Learned Reward Functions
  3. 2026When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient
  4. 2024The Distributional Reward Critic Framework for Reinforcement Learning Under Perturbed Rewards
  5. 2025A Systematic Analysis of Base Model Choice for Reward Modeling
  6. 2024Improving Reinforcement Learning from Human Feedback Using Contrastive Rewards
  7. 2024To the Max: Reinventing Reward in Reinforcement Learning
  8. 2025Reinforcement Learning from Multi-level and Episodic Human Feedback