mlsanityData to 14 Oct 2026

Multi-objective Reinforcement Learning With Augmented States Requires Rewards After Deployment

arXiv 2604.15757 ↗First posted 17 Apr 2026cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization

Abstract (arXiv)

This research note identifies a previously overlooked distinction between multi-objective reinforcement learning (MORL), and more conventional single-objective reinforcement learning (RL). It has previously been noted that the optimal policy for an MORL agent with a non-linear utility function is required to be conditioned on both the current environmental state and on some measure of the previously accrued reward. This is generally implemented by concatenating the observed state of the environment with the discounted sum of previous rewards to create an augmented state. While augmented states have been widely-used in the MORL literature, one implication of their use has not previously been reported -- namely that they require the agent to have continued access to the reward signal (or a proxy thereof) after deployment, even if no further learning is required. This note explains why this is the case, and considers the practical repercussions of this requirement.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 15 of its 30 references that are in the map. First posted on arXiv 17 Apr 2026.

Builds on

15 of 30 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024An Empirical Investigation of Value-Based Multi-objective Reinforcement Learning for Stochastic Environments6 Jan

Other references · 14

  1. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  2. 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 256
  3. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 242
  4. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 213
  5. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 122
  6. 2018End-to-end Active Object Tracking and Its Real-world Deployment via Reinforcement Learningcited by 35
  7. 2023Multi-Objective Reinforcement Learning Based on Decomposition: A Taxonomy and Frameworkcited by 31
  8. 2025Reward Models in Deep Reinforcement Learning: A Surveycited by 25
  9. 2024Provable Partially Observable Reinforcement Learning with Privileged Informationcited by 13
  10. 2021Joint Optimization of Multi-Objective Reinforcement Learning with Policy Gradient Based Algorithmcited by 12
  11. 2024Monitored Markov Decision Processescited by 10
  12. 2025A Comprehensive Review of Multi-Agent Reinforcement Learning in Video Gamescited by 9
2 more
  1. 2023Multi-objective Reinforcement Learning with Nonlinear Preferences: Provable Approximation for Maximizing Expected Scalarized Returncited by 7
  2. 2022Actively Learning Costly Reward Functions for Reinforcement Learningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Utility-Based Reinforcement Learning: Unifying Single-objective and Multi-objective Reinforcement Learning
  2. 2026A Reward-Free Viewpoint on Multi-Objective Reinforcement Learning
  3. 2025On Generalization Across Environments In Multi-Objective Reinforcement Learning
  4. 2025Preference-based Multi-Objective Reinforcement Learning
  5. 2023Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RL
  6. 2024C-MORL: Multi-Objective Reinforcement Learning through Efficient Discovery of Pareto Front
  7. 2024Demonstration Guided Multi-Objective Reinforcement Learning
  8. 2025Benchmarking MOEAs for solving continuous multi-objective RL problems