mlsanityData to 13 Oct 2026

The Distributional Reward Critic Framework for Reinforcement Learning Under Perturbed Rewards

arXiv 2401.05710 ↗First posted 11 Jan 2024cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

The reward signal plays a central role in defining the desired behaviors of agents in reinforcement learning (RL). Rewards collected from realistic environments could be perturbed, corrupted, or noisy due to an adversary, sensor error, or because they come from subjective human feedback. Thus, it is important to construct agents that can learn under such rewards. Existing methodologies for this problem make strong assumptions, including that the perturbation is known in advance, clean rewards are accessible, or that the perturbation preserves the optimal policy. We study a new, more general, class of unknown perturbations, and introduce a distributional reward critic framework for estimating reward distributions and perturbations during training. Our proposed methods are compatible with any RL algorithm. Despite their increased generality, we show that they achieve comparable or better rewards than existing methods in a variety of environments, including those with clean rewards. Under the challenging and generalized perturbations we study, we win/tie the highest return in 44/48 tested settings (compared to 11/48 for the best baseline). Our results broaden and deepen our ability to perform RL in reward-perturbed environments.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 16 of its 26 references that are in the map. First posted on arXiv 11 Jan 2024.

Builds on

16 of 26 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 15

  1. 2022Training language models to follow instructions with human feedbackcited by 15,856
  2. 2017Proximal Policy Optimization Algorithmscited by 14,592
  3. 2015Continuous control with deep reinforcement learningcited by 3,290
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,221
  5. 2017A Distributional Perspective on Reinforcement Learningcited by 745
  6. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 465
  7. 2017Robust Adversarial Reinforcement Learningcited by 440
  8. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 363
  9. 2017Inverse Reward Designcited by 248
  10. 2017Robust Deep Reinforcement Learning with Adversarial Attackscited by 153
  11. 2020Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learningcited by 81
  12. 2018Reinforcement Learning with Perturbed Rewardscited by 76
3 more
  1. 2017Reinforcement Learning with a Corrupted Reward Channelcited by 68
  2. 2018An Analysis of Categorical Distributional Reinforcement Learningcited by 68
  3. 2018Reward Estimation for Variance Reduction in Deep Reinforcement Learningcited by 17

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Reward Models in Deep Reinforcement Learning: A Survey
  2. 2025Distributional Inverse Reinforcement Learning
  3. 2026When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient
  4. 2024The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret
  5. 2024The Value of Reward Lookahead in Reinforcement Learning
  6. 2025Learning a Pessimistic Reward Model in RLHF
  7. 2026Binary Rewards and Reinforcement Learning: Fundamental Challenges
  8. 2025Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers