mlsanityData to 14 Oct 2026

Safety, Security, and Cognitive Risks in World Models

arXiv 2604.01346 ↗First posted 1 Apr 2026cs.CR cs.AI cs.LGcited by 3

On the map: World models → World model security and GUI agents

also in 1 more
Abstract (arXiv)

World models - learned internal simulators of environment dynamics - are rapidly becoming foundational to autonomous decision-making in robotics, autonomous vehicles, and agentic AI. By predicting future states in compressed latent spaces, they enable sample-efficient planning and long-horizon imagination without direct environment interaction. Yet this predictive power introduces a distinctive set of safety, security, and cognitive risks. Adversaries can corrupt training data, poison latent representations, and exploit compounding rollout errors to cause significant degradation in safety-critical deployments. At the alignment layer, world model-equipped agents are more capable of goal misgeneralisation, deceptive alignment, and reward hacking. At the human layer, authoritative world model predictions foster automation bias, miscalibrated trust, and planning hallucination. This paper surveys the world model landscape; introduces formal definitions of trajectory persistence and representational risk; presents a five-profile attacker taxonomy; and develops a unified threat model drawing on MITRE ATLAS and the OWASP LLM Top 10. We provide an empirical proof-of-concept demonstrating trajectory-persistent adversarial attacks on a GRU-based RSSM (A1 = 2.26× amplification, -59.5% reward reduction under adversarial fine-tuning), validate architecture-dependence via a stochastic RSSM proxy (A1 = 0.65×), and probe a real DreamerV3 checkpoint (non-zero action drift confirmed). We propose interdisciplinary mitigations spanning adversarial hardening, alignment engineering, NIST AI RMF and EU AI Act governance, and human-factors design, arguing that world models require the same rigour as flight-control software or medical devices.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in World model security and GUI agents, one of 11 directions in World models. Builds on 54 of its 62 references that are in the map. First posted on arXiv 1 Apr 2026.

Builds on

54 of 62 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026When World Models Dream Wrong: Physical-Conditioned Adversarial Attacks against World Models21 Feb

Other references · 53

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  2. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  4. 2018World Modelscited by 1,513
  5. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
  6. 2023Mastering Diverse Domains through World Modelscited by 1,253
  7. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  8. 2024Genie: Generative Interactive Environmentscited by 747
  9. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  10. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  11. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  12. 2017Provable defenses against adversarial examples via the convex outer adversarial polytopecited by 567
38 more
  1. 2018Certified Robustness to Adversarial Examples with Differential Privacycited by 468
  2. 2023Learning Interactive Real-World Simulatorscited by 458
  3. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  4. 2019Provably Robust Deep Learning via Adversarially Trained Smoothed Classifierscited by 330
  5. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 261
  6. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 258
  7. 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
  8. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 238
  9. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
  10. 2022Model-Based Imitation Learning for Urban Drivingcited by 150
  11. 2020Randomized Smoothing of All Shapes and Sizescited by 147
  12. 2021AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehiclescited by 137
  13. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 123
  14. 2025A Comprehensive Survey on World Models for Embodied AIcited by 68
  15. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 67
  16. 2025A Survey of World Models for Autonomous Drivingcited by 60
  17. 2018Algorithms that Remember: Model Inversion Attacks and Data Protection Lawcited by 55
  18. 2023SafeDreamer: Safe Reinforcement Learning with World Modelscited by 54
  19. 2025Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Roboticscited by 51
  20. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 50
  21. 2024Privacy Leakage on DNNs: A Survey of Model Inversion Attacks and Defensescited by 30
  22. 2020Mitigating Value Hallucination in Dyna Planning via Multistep Predecessor Modelscited by 28
  23. 2022Lyapunov Density Models: Constraining Distribution Shift in Learning-Based Controlcited by 28
  24. 2025Data Poisoning in Deep Learning: A Surveycited by 20
  25. 2023Adversarial Robust Deep Reinforcement Learning Requires Redefining Robustnesscited by 19
  26. 2025A Survey on Model Extraction Attacks and Defenses for Large Language Modelscited by 19
  27. 2026Current Agents Fail to Leverage World Model as Tool for Foresightcited by 17
  28. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  29. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
  30. 2021Deep Reinforcement Learning Policies Learn Shared Adversarial Features Across MDPscited by 12
  31. 2025Four Principles for Physically Interpretable World Modelscited by 10
  32. 2024Robust Deep Reinforcement Learning with Adaptive Adversarial Perturbations in Action Spacecited by 9
  33. 2024World Models: The Safety Perspectivecited by 9
  34. 2023Detecting Adversarial Directions in Deep Reinforcement Learning to Make Robust Decisionscited by 7
  35. 2024Understanding and Diagnosing Deep Reinforcement Learningcited by 6
  36. 2025Misalignment or misuse? The AGI alignment tradeoffcited by 4
  37. 2024Risk Alignment in Agentic AI Systemscited by 3
  38. 2025Bounding Distributional Shifts in World Modeling through Novelty Detectioncited by 2

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026ARB4WM: An Adversarial Robustness Benchmark for World Models in Continuous Control
  2. 2025The Safety Challenge of World Models for Embodied AI Agents: A Review
  3. 2026BadWAM: When World-Action Models Dream Right but Act Wrong
  4. 2026False Prophets: On the Security of World Models in Agentic Systems
  5. 2026Rethinking World Models for Safety-Critical Embodied Systems
  6. 2026Targeting World Models to Compromise Robot Learning Pipelines
  7. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents
  8. 2026CorrRisk-WM: Corridor-Conditioned Risk World Modeling for Safety-Critical Trajectory Planning