mlsanityData to 14 Oct 2026

Hindsight Preference Replay Improves Preference-Conditioned Multi-Objective Reinforcement Learning

arXiv 2601.11604 ↗First posted 8 Jan 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Multi-objective reinforcement learning (MORL) enables agents to optimize vector-valued rewards while respecting user preferences. CAPQL, a preference-conditioned actor-critic method, achieves this by conditioning on weight vectors w and restricts data usage to the specific preferences under which it was collected, leaving off-policy data from other preferences unused. We introduce Hindsight Preference Replay (HPR), a simple and general replay augmentation strategy that retroactively relabels stored transitions with alternative preferences. This densifies supervision across the preference simplex without altering the CAPQL architecture or loss functions. Evaluated on six MO-Gymnasium locomotion tasks at a fixed 300000-step budget using expected utility (EUM), hypervolume (HV), and sparsity, HPR-CAPQL improves HV in five of six environments and EUM in four of six. On mo-humanoid-v5, for instance, EUM rises from 323!±!125 to 1613!±!464 and HV from 0.52M to 9.63M, with strong statistical support. mo-halfcheetah-v5 remains a challenging exception where CAPQL attains higher HV at comparable EUM. We report final summaries and Pareto-front visualizations across all tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 7 of its 18 references that are in the map. First posted on arXiv 8 Jan 2026.

Builds on

7 of 18 references.

References · 7

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2017Hindsight Experience Replaycited by 1,136
  3. 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 256
  4. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 242
  5. 2025Preference-Driven Multi-Objective Combinatorial Optimization with Conditional Computationcited by 6
  6. 2024Optimizing Deep Reinforcement Learning for Adaptive Robotic Arm Controlcited by 2
  7. 2018Advances in Experience Replaycited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026PCHC: Enabling Preference Conditioned Humanoid Control via Multi-Objective Reinforcement Learning
  2. 2026On Preference Coverage Collapse from Hindsight Relabeling in Multi-Objective Reinforcement Learning
  3. 2023A State Augmentation based approach to Reinforcement Learning from Human Preferences
  4. 2024Hindsight PRIORs for Reward Learning from Human Preferences
  5. 2025Fusing Rewards and Preferences in Reinforcement Learning
  6. 2026MAPL: Multi-Objective Preference Learning for Robot Locomotion ↗
  7. 2025Residual Reward Models: Leveraging Prior Knowledge for Efficient Preference-based Reinforcement Learning in Robotics
  8. 2026Optimizing Neurorobot Policy under Limited Demonstration Data through Preference Regret