Across Viewpoints and Time: Trajectory-Generalizable Physical Attacks on World Models and Video Understanding Models
On the map: World models → Video-generation world models and simulators
Abstract (arXiv)
World models and video understanding models are increasingly deployed in dynamic visual environments, yet their robustness to physical adversarial threats remains largely underexplored. We study whether a single physical adversarial texture can mislead the victim model across different real-world scenarios where observing viewpoints vary along uncontrolled object--camera trajectories. We identify that heterogeneous video semantics and trajectories induce poorly aligned gradients for optimizing the adversarial texture. To address that, we propose the Trajectory-aware adversarial Alignment and Refinement Attack (TARA), which incorporates trajectory-aware optimization to capture the evolving geometric variations of a printed texture under real-world scenarios. To improve cross-instance optimization compatibility, TARA incorporates trajectory-aware contrastive alignment to encourage attacked representations from diverse video--trajectory instances to approach a shared adversarial region and then performs trajectory-aware task refinement to translate the aligned representation into task-level prediction errors. For reproducible evaluation, we develop a geometry-grounded tracking pipeline that provides physically plausible adversarial regions with temporally consistent trajectories on EPIC-KITCHENS-100 and Something-Something V2. We further collect real-world print-and-recapture videos to validate attack effectiveness under physical deployment conditions. Together, the annotated benchmarks and real-world captured data form a reproducible testbed for future evaluation. Notably, our results reveal that a single printable adversarial texture can persistently disrupt world models and video understanding models under physical conditions, highlighting the fundamental security challenge.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 41 of its 48 references that are in the map. First posted on arXiv 11 Oct 2026.
Builds on
41 of 48 references.
References · 41
- 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
- 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
- 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
- 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
- 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
- 2020Rescaling Egocentric Visioncited by 540
- 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
- 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
- 2017Adversarial Patchcited by 517
- 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 386
- 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 317
- 2023RoMa: Robust Dense Feature Matchingcited by 302
29 more
- 2021Anticipative Video Transformercited by 200
- 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 146
- 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 117
- 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 105
- 2024Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Roboticscited by 73
- 2018Sparse Adversarial Perturbations for Videoscited by 57
- 2022Patch-Fool: Are Vision Transformers Always Robust Against Adversarial Perturbations?cited by 49
- 2025TAPNext: Tracking Any Point (TAP) as Next Token Predictioncited by 37
- 2020Over-the-Air Adversarial Flickering Attacks against Video Recognition Networkscited by 36
- 2019APRICOT: A Dataset of Physical Adversarial Attacks on Object Detectioncited by 31
- 2024Transferable Adversarial Attacks on SAM and Its Downstream Modelscited by 21
- 2022REAP: A Large-Scale Realistic Adversarial Patch Benchmarkcited by 18
- 2024Diffusion Policy Attacker: Crafting Adversarial Attacks for Diffusion-based Policiescited by 18
- 2025When Robots Obey the Patch: Universal Transferable Patch Attacks on Vision-Language-Action Modelscited by 13
- 2026When World Models Dream Wrong: Physical-Conditioned Adversarial Attacks against World Modelscited by 13
- 2026DiLA: Disentangled Latent Action World Modelscited by 10
- 2023Breaking Temporal Consistency: Generating Video Universal Adversarial Perturbations Using Image Modelscited by 9
- 2025UV-Attack: Physical-World Adversarial Attacks for Person Detection via Dynamic-NeRF-based UV Mappingcited by 9
- 2025Cross-Modal Transferable Image-to-Video Attack on Video Quality Metricscited by 9
- 2025From Pretrain to Pain: Adversarial Vulnerability of Video Foundation Models Without Task Knowledgecited by 9
- 20253D Gaussian Splatting Driven Multi-View Robust Physical Adversarial Camouflage Generationcited by 7
- 2026WMAttack: Automated Attack Search for Adversarial Evaluation of World-Model Agentscited by 7
- 2026TAPNext++: What's Next for Tracking Any Point (TAP)?cited by 6
- 2025PapMOT: Exploring Adversarial Patch Attack against Multiple Object Trackingcited by 4
- 2026Partially Observable Adversarial Patch Attacks on Vision-Language-Action Models in Roboticscited by 4
- 2026Tex3D: Objects as Attack Surfaces via Adversarial 3D Textures for Vision-Language-Action Modelscited by 3
- 2026Structure Abstraction and Generalization in a Hippocampal-Entorhinal Inspired World Modelcited by 1
- 2026Targeting World Models to Compromise Robot Learning Pipelinescited by 1
- 2026UniTexture: Cross-Task Universal Adversarial Textures for Vision-Language-Action Modelscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026BadWorld: Adversarial Attacks on World Models
- 2026A Survey on Adversarial Attacks and Defenses for Diffusion Models Across Multiple Modalities
- 2025Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modeling
- 2026Still Camouflage, Moving Illusion: View-Induced Trajectory Manipulation in Autonomous Driving
- 2026WorldOlympiad: Can Your World Model Survive a Triathlon?
- 2026VIGOR: VIdeo Geometry-Oriented Reward for Temporal Generative Alignment
- 2026PROWL: Prioritized Regret-Driven Optimization for World Model Learning
- 2026UniNav: A Unified World-Action Diffusion Model for Visual Navigation