Learning Safe Agent Behaviour from Human Preferences and Justifications via World Models

arXiv 2607.13172 ↗First posted 14 Jul 2026cs.AI cs.LGcited by 0

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

We address the problem of safely training an agent policy and deploying a good and safe policy, in settings where the environment dynamics are unknown and no suitable reward function is available. In the context of safety-critical environments, we consider traditional reinforcement learning impractical and resort to the resource of human input. We introduce DROPJ, a human-centred method for both safe training and deployment. We first learn a world model (a learned simulator) from a dataset of prior real-world trajectories. A human then plays the game in this learned simulator to extract several informative simulated trajectories. From these, we sample pairs of simulated trajectory segments and elicit from a human their preference over these segments, as well as a reason (justification) for their choice. We then train a reward model from these justified preferences and use it, together with the world model, to directly deploy the agent using model predictive control. Running real-user experiments, we find that generating informative simulated trajectories from a user significantly reduces the computational cost during training compared to other strategies, and can also improve the performance during deployment. In the context of training within a learned simulator, we show that the use of preferences rather than other types of feedback substantially improves the performance during deployment. We further demonstrate that safety justifications accompanying preferences can significantly enhance safety or prioritise user-prescribed aspects of safety associated with them during deployment.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 43 of its 77 references that are in the map. First posted on arXiv 14 Jul 2026.

Builds on

43 of 77 references.

Closest ancestors · 5

Same direction, newest first

2023SafeDreamer: Safe Reinforcement Learning with World Models14 Jul
2022Constrained Policy Optimization via Bayesian World Models24 Jan
2020Mastering Atari with Discrete World Models5 Oct
2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec
2018Recurrent World Models Facilitate Policy Evolution4 Sep

Other references · 38

2022Training language models to follow instructions with human feedbackcited by 15,712
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
2017Deep reinforcement learning from human preferencescited by 3,846
2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
2017Constrained Policy Optimizationcited by 928
2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
2018Scalable agent alignment via reward modeling: a research directioncited by 438
2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278

26 more

2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 256
2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
2017Trial without Error: Towards Safe Reinforcement Learning via Human Interventioncited by 102
2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 96
2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 94
2018Batch Active Preference-Based Learning of Reward Functionscited by 87
2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77
2020Safe Reinforcement Learning via Curriculum Inductioncited by 73
2019Learning Human Objectives by Evaluating Hypothetical Behaviorcited by 41
2021Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violationscited by 36
2022Towards Safe Reinforcement Learning with a Safety Editor Policycited by 28
2023Safe Exploration in Reinforcement Learning: A Generalized Formulation and Algorithmscited by 27
2019Risk-Aware Active Inverse Reinforcement Learningcited by 26
2023Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Modelscited by 20
2023A Human-Centered Safe Robot Reinforcement Learning Framework with Interactive Behaviorscited by 13
2021Safe Policy Optimization with Local Generalized Linear Function Approximationscited by 10
2018Efficiently Combining Human Demonstrations and Interventions for Safe Training of Autonomous Systems in Real-Timecited by 7
2019Parenting: Safe Reinforcement Learning from Human Inputcited by 6
2021Diversity-based Trajectory and Goal Selection with Hindsight Experience Replaycited by 6
2022Safe Deep RL in 3D Environments using Human Feedbackcited by 5
2024Safe Reinforcement Learning with Free-form Natural Language Constraints and Pre-Trained Language Modelscited by 4
2022Multi-trainer Interactive Reinforcement Learning Systemcited by 2
2026SHAPO: Sharpness-Aware Policy Optimization for Safe Explorationcited by 2
2024Rejecting Hallucinated State Targets during Planningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way