mlsanityData to 7 Oct 2026

Internally Rewarded Reinforcement Learning

arXiv 2302.00270 ↗First posted 1 Feb 2023cs.LG cs.AIcited by 14

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

We study a class of reinforcement learning problems where the reward signals for policy learning are generated by an internal reward model that is dependent on and jointly optimized with the policy. This interdependence between the policy and the reward model leads to an unstable learning process because reward signals from an immature reward model are noisy and impede policy learning, and conversely, an under-optimized policy impedes reward estimation learning. We call this learning setting textitInternally Rewarded Reinforcement Learning (IRRL) as the reward is not provided directly by the environment but textitinternally by a reward model. In this paper, we formally formulate IRRL and present a class of problems that belong to IRRL. We theoretically derive and empirically analyze the effect of the reward function in IRRL and based on these analyses propose the clipped linear reward function. Experimental results show that the proposed reward function can consistently stabilize the training process by reducing the impact of reward noise, which leads to faster convergence and higher performance compared with baselines in diverse tasks.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 34 of its 59 references that are in the map. First posted on arXiv 1 Feb 2023.

Builds on

34 of 59 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  7. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  8. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  9. 2018MINE: Mutual Information Neural Estimationcited by 680
  10. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 504
  11. 2014Recurrent Models of Visual Attentioncited by 344
  12. 2017Embodied Question Answeringcited by 335
21 more
  1. 2017Adversarial Attacks on Neural Network Policiescited by 314
  2. 2016Variational Intrinsic Controlcited by 262
  3. 2017IQA: Visual Question Answering in Interactive Environmentscited by 154
  4. 2018Reinforcement Learning with Perturbed Rewardscited by 76
  5. 2017Reinforcement Learning with a Corrupted Reward Channelcited by 68
  6. 2021Diffusion-Based Representation Learningcited by 57
  7. 2013Equitability, mutual information, and the maximal information coefficientcited by 45
  8. 2020When Optimizing f-divergence is Robust with Label Noisecited by 44
  9. 2021Learning more skills through optimistic explorationcited by 37
  10. 2019Saccader: Improving Accuracy of Hard Attention Models for Visioncited by 35
  11. 2017Dynamic Computational Time for Visual Attentioncited by 23
  12. 2019Learning Representations by Maximizing Mutual Information in Variational Autoencoderscited by 20
  13. 2020f-GAIL: Learning f-Divergence for Generative Adversarial Imitation Learningcited by 20
  14. 2017Learning to Skim Textcited by 18
  15. 2020MQA: Answering the Question via Robotic Manipulationcited by 11
  16. 2022Consistency driven Sequential Transformers Attention Model for Partially Observable Scenescited by 11
  17. 2020Towards Embodied Scene Descriptioncited by 6
  18. 2018EmoRL: Continuous Acoustic Emotion Classification using Deep Reinforcement Learningcited by 4
  19. 2021Robotic Occlusion Reasoning for Efficient Object Existence Predictioncited by 4
  20. 2015Recurrent Spatial Transformer Networkscited by 3
  21. 2022A Unified f-divergence Framework Generalizing VAE and GANcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 4; 2025: 4; 2026: 1.

14 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  3. 2023Chat with the Environment: Interactive Multimodal Perception Using Large Language Modelscited by 42
  4. 2025Curriculum-RLAIF: Curriculum Alignment with Reinforcement Learning from AI Feedbackcited by 30
  5. 2024Can a MISL Fly? Analysis and Ingredients for Mutual Information Skill Learningcited by 18
  6. 2024Large Language Models for Orchestrating Bimanual Robotscited by 10
  7. 2024Agentic Skill Discoverycited by 5
  8. 2024Scene Exploration by Vision-Language Modelscited by 5
  9. 2023Sample-efficient Real-time Planning with Curiosity Cross-Entropy Method and Contrastive Learningcited by 3
  10. 2023Causal State Distillation for Explainable Reinforcement Learningcited by 2
  11. 2025PersRM-R1: Enhance Personalized Reward Modeling with Reinforcement Learningcited by 2
  12. 2025Apple: Toward General Active Perception via Reinforcement Learningcited by 1
2 more
  1. 2025Distributional Inverse Reinforcement Learningcited by 0
  2. 2026When Does External Guidance Help LLM Reasoning? A Bias-Variance Theory of Guidance-Augmented GRPOcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reward Models in Deep Reinforcement Learning: A Survey
  2. 2023BC-IRL: Learning Generalizable Reward Functions from Demonstrations
  3. 2026Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policyness
  4. 2026When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient
  5. 2026Inverse Reinforcement Learning without an Optimal Demonstrator: A Feasible Reward Set Approach
  6. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking
  7. 2026Reward-Conditioned Reinforcement Learning
  8. 2025Which Rewards Matter? Reward Selection for Reinforcement Learning under Limited Feedback