mlsanityData to 7 Oct 2026

Reward-Conditioned Reinforcement Learning

arXiv 2603.05066 ↗First posted 5 Mar 2026cs.LGcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Single-task RL agents are typically trained under a fixed reward function, which limits their robustness to reward misspecification and their ability to adapt to changing preferences. We introduce Reward-Conditioned Reinforcement Learning (RCRL), an off-policy method that conditions agents on reward parameterizations while collecting experience under a single nominal objective. By recomputing counterfactual rewards from shared replay data, RCRL exposes the agent to multiple reward objectives without additional environment interaction, connecting single-task RL with ideas from multi-objective and multi-task learning. Across single-task, multi-task, and vision-based benchmarks, RCRL improves sample efficiency under the nominal reward parameterization, enables efficient adaptation to new parameterizations, and supports zero-shot behavioral adjustment at deployment. Our results show that RCRL provides a scalable mechanism for learning robust, steerable policies without sacrificing the simplicity of single-task training.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 80 of its 98 references that are in the map. First posted on arXiv 5 Mar 2026.

Builds on

80 of 98 references.

References · 80

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  7. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  8. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  9. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  10. 2017Hindsight Experience Replaycited by 1,096
  11. 2018DeepMind Control Suitecited by 935
  12. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
38 more
  1. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  2. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  3. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  4. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  5. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  6. 2019Solving Rubik's Cube with a Robot Handcited by 699
  7. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  8. 2018Learning Dexterous In-Hand Manipulationcited by 643
  9. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  10. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  11. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  12. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  13. 2020Reinforcement Learning with Augmented Datacited by 452
  14. 2019Learning agile and dynamic motor skills for legged robotscited by 402
  15. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  16. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 395
  17. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  18. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  19. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  20. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  21. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  22. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  23. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  24. 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 240
  25. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  26. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 229
  27. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 227
  28. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
  29. 2015Actor-Mimic: Deep Multitask and Transfer Reinforcement Learningcited by 176
  30. 2019A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptationcited by 166
  31. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 157
  32. 2018Multi-task Deep Reinforcement Learning with PopArtcited by 138
  33. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  34. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 124
  35. 2021Learning One Representation to Optimize All Rewardscited by 110
  36. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
  37. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 104
  38. 2018Universal Successor Features Approximatorscited by 102

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026FlowR2A: Learning Reward-to-Action Distribution for Multimodal Driving Planningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Bayesian Reparameterization of Reward-Conditioned Reinforcement Learning with Energy-based Models
  2. 2026LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback
  3. 2026Improving Zero-Shot Offline RL via Behavioral Task Sampling
  4. 2026PHIRL: Aligning Learned Rewards with Task Progress for Inverse Reinforcement Learning
  5. 2025Reward Models in Deep Reinforcement Learning: A Survey
  6. 2026Scalable Multi-Task Inverse Reinforcement Learning
  7. 2026Reinforcement Learning with Decomposed Subtasks
  8. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training