Safety, Security, and Cognitive Risks in World Models
On the map: World models → World model security and GUI agents
also in 1 more
Abstract (arXiv)
World models - learned internal simulators of environment dynamics - are rapidly becoming foundational to autonomous decision-making in robotics, autonomous vehicles, and agentic AI. By predicting future states in compressed latent spaces, they enable sample-efficient planning and long-horizon imagination without direct environment interaction. Yet this predictive power introduces a distinctive set of safety, security, and cognitive risks. Adversaries can corrupt training data, poison latent representations, and exploit compounding rollout errors to cause significant degradation in safety-critical deployments. At the alignment layer, world model-equipped agents are more capable of goal misgeneralisation, deceptive alignment, and reward hacking. At the human layer, authoritative world model predictions foster automation bias, miscalibrated trust, and planning hallucination. This paper surveys the world model landscape; introduces formal definitions of trajectory persistence and representational risk; presents a five-profile attacker taxonomy; and develops a unified threat model drawing on MITRE ATLAS and the OWASP LLM Top 10. We provide an empirical proof-of-concept demonstrating trajectory-persistent adversarial attacks on a GRU-based RSSM (A1 = 2.26× amplification, -59.5% reward reduction under adversarial fine-tuning), validate architecture-dependence via a stochastic RSSM proxy (A1 = 0.65×), and probe a real DreamerV3 checkpoint (non-zero action drift confirmed). We propose interdisciplinary mitigations spanning adversarial hardening, alignment engineering, NIST AI RMF and EU AI Act governance, and human-factors design, arguing that world models require the same rigour as flight-control software or medical devices.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in World model security and GUI agents, one of 11 directions in World models. Builds on 54 of its 62 references that are in the map. First posted on arXiv 1 Apr 2026.
Builds on
54 of 62 references.
Closest ancestors · 1
Same direction, newest first
- 2026When World Models Dream Wrong: Physical-Conditioned Adversarial Attacks against World Models21 Feb
Other references · 53
- 2014Explaining and Harnessing Adversarial Examplescited by 7,364
- 2020Shortcut Learning in Deep Neural Networkscited by 1,969
- 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
- 2018World Modelscited by 1,513
- 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
- 2023Mastering Diverse Domains through World Modelscited by 1,253
- 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
- 2024Genie: Generative Interactive Environmentscited by 747
- 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
- 2020MOPO: Model-based Offline Policy Optimizationcited by 636
- 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
- 2017Provable defenses against adversarial examples via the convex outer adversarial polytopecited by 567
38 more
- 2018Certified Robustness to Adversarial Examples with Differential Privacycited by 468
- 2023Learning Interactive Real-World Simulatorscited by 458
- 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
- 2019Provably Robust Deep Learning via Adversarially Trained Smoothed Classifierscited by 330
- 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 261
- 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 258
- 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
- 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 238
- 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
- 2022Model-Based Imitation Learning for Urban Drivingcited by 150
- 2020Randomized Smoothing of All Shapes and Sizescited by 147
- 2021AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehiclescited by 137
- 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 123
- 2025A Comprehensive Survey on World Models for Embodied AIcited by 68
- 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 67
- 2025A Survey of World Models for Autonomous Drivingcited by 60
- 2018Algorithms that Remember: Model Inversion Attacks and Data Protection Lawcited by 55
- 2023SafeDreamer: Safe Reinforcement Learning with World Modelscited by 54
- 2025Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Roboticscited by 51
- 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 50
- 2024Privacy Leakage on DNNs: A Survey of Model Inversion Attacks and Defensescited by 30
- 2020Mitigating Value Hallucination in Dyna Planning via Multistep Predecessor Modelscited by 28
- 2022Lyapunov Density Models: Constraining Distribution Shift in Learning-Based Controlcited by 28
- 2025Data Poisoning in Deep Learning: A Surveycited by 20
- 2023Adversarial Robust Deep Reinforcement Learning Requires Redefining Robustnesscited by 19
- 2025A Survey on Model Extraction Attacks and Defenses for Large Language Modelscited by 19
- 2026Current Agents Fail to Leverage World Model as Tool for Foresightcited by 17
- 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
- 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
- 2021Deep Reinforcement Learning Policies Learn Shared Adversarial Features Across MDPscited by 12
- 2025Four Principles for Physically Interpretable World Modelscited by 10
- 2024Robust Deep Reinforcement Learning with Adaptive Adversarial Perturbations in Action Spacecited by 9
- 2024World Models: The Safety Perspectivecited by 9
- 2023Detecting Adversarial Directions in Deep Reinforcement Learning to Make Robust Decisionscited by 7
- 2024Understanding and Diagnosing Deep Reinforcement Learningcited by 6
- 2025Misalignment or misuse? The AGI alignment tradeoffcited by 4
- 2024Risk Alignment in Agentic AI Systemscited by 3
- 2025Bounding Distributional Shifts in World Modeling through Novelty Detectioncited by 2
50 most cited of 53 All 53 (JSON)
Built on by
Citing papers per year: 2026: 3.
3 papers.
- 2026Security of World-Model-Based Embodied AI: A Lifecycle of Threats, Defenses, and Evaluationcited by 2
- 2026BadDreamer: Transferable Backdoor Attacks against Video World Models for Autonomous Drivingcited by 1
- 2026When the World Lies: Backdoor Attacks on Latent World Models for Downstream Controlcited by 0
- 2026When the World Lies: Backdoor Attacks on Latent World Models for Downstream Control14 Sep 2026
- 2026Security of World-Model-Based Embodied AI: A Lifecycle of Threats, Defenses, and Evaluation30 Jul 2026
- 2026BadDreamer: Transferable Backdoor Attacks against Video World Models for Autonomous Driving19 Jun 2026
Similar, not cited
Close in wording, no citation either way
- 2026ARB4WM: An Adversarial Robustness Benchmark for World Models in Continuous Control
- 2025The Safety Challenge of World Models for Embodied AI Agents: A Review
- 2026BadWAM: When World-Action Models Dream Right but Act Wrong
- 2026False Prophets: On the Security of World Models in Agentic Systems
- 2026Rethinking World Models for Safety-Critical Embodied Systems
- 2026Targeting World Models to Compromise Robot Learning Pipelines
- 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents
- 2026CorrRisk-WM: Corridor-Conditioned Risk World Modeling for Safety-Critical Trajectory Planning