# ACWM-Phys: Investigating Generalized Physical Interaction in Action-Conditioned Video World Models

arXiv: https://arxiv.org/abs/2605.08567 · HTML: https://mlsanity.com/paper/2605.08567/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Latent predictive world models, one of 11 directions in World models.
- Builds on 27 of its 33 references that are in the map.
- First posted on arXiv 9 May 2026.

## Placed in

- World models / [Latent predictive world models](https://mlsanity.com/wm/latent-predictive-world-models/)

## Builds on (most cited first, 27 of 27)

- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024)
- [Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion](https://mlsanity.com/paper/2506.08009/) (2025)
- [World Action Models are Zero-shot Policies](https://mlsanity.com/paper/2602.15922/) (2026)
- [Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids](https://mlsanity.com/paper/1810.01566/) (2018)
- [How Far is Video Generation from World Model: A Physical Law Perspective](https://mlsanity.com/paper/2411.02385/) (2024)
- [WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling](https://mlsanity.com/paper/2512.14614/) (2025)
- [Ctrl-World: A Controllable Generative World Model for Robot Manipulation](https://mlsanity.com/paper/2510.10125/) (2025)
- [Do generative video models understand physical principles?](https://mlsanity.com/paper/2501.09038/) (2025)
- [Training Agents Inside of Scalable World Models](https://mlsanity.com/paper/2509.24527/) (2025)
- [Rapid Exploration for Open-World Navigation with Latent Goal Models](https://mlsanity.com/paper/2104.05859/) (2021)
- [WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generation](https://mlsanity.com/paper/2503.08153/) (2025)
- [Vid2World: Crafting Video Diffusion Models to Interactive World Models](https://mlsanity.com/paper/2505.14357/) (2025)
- [PISA Experiments: Exploring Physics Post-Training for Video Diffusion Models by Watching Stuff Drop](https://mlsanity.com/paper/2503.09595/) (2025)
- [WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL](https://mlsanity.com/paper/2602.13977/) (2026)
- [Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systems](https://mlsanity.com/paper/2504.02918/) (2025)
- [Solaris: Building a Multiplayer Video World Model in Minecraft](https://mlsanity.com/paper/2602.22208/) (2026)
- [NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamics](https://mlsanity.com/paper/2509.21309/) (2025)
- [What about gravity in video generation? Post-Training Newton's Laws with Verifiable Rewards](https://mlsanity.com/paper/2512.00425/) (2025)
- [BridgeV2W: Bridging Video Generation Models to Embodied World Models via Embodiment Masks](https://mlsanity.com/paper/2602.03793/) (2026)
- [Walk through Paintings: Egocentric World Models from Internet Priors](https://mlsanity.com/paper/2601.15284/) (2026)
- [ProPhy: Progressive Physical Alignment for Dynamic World Simulation](https://mlsanity.com/paper/2512.05564/) (2025)

## Built on by (most cited first, 10 of 10)

- [Cosmos 3: Omnimodal World Models for Physical AI](https://mlsanity.com/paper/2606.02800/) (2026)
- [WorldExam: Benchmarking World Models from Apparent Appearance to Inherent Reactivity](https://mlsanity.com/paper/2608.02603/) (2026)
- [Causally Debiased Latent Action Model for Embodied Action Conditioned World Models](https://mlsanity.com/paper/2607.09185/) (2026)
- [WorldSimProbe: Diagnosing Simulator Faithfulness in Action-Conditioned World Models for Embodied Manipulation](https://mlsanity.com/paper/2608.09298/) (2026)
- [Semigroup-JEPA: Latent Dynamics Consistency for Zero-Shot Physics Generalization](https://mlsanity.com/paper/2609.10464/) (2026)
- [CausalWM: Causal Chain-of-Thought Reasoning for Embodied World Model](https://mlsanity.com/paper/2609.23184/) (2026)
- [WorldBagel: Uncovering the Power of Unified Multimodal Models for Vision-Language-Action-World Modeling](https://mlsanity.com/paper/2607.03461/) (2026)
- [PhysAI-Bench: A Benchmark for LLM-Based Agentic Decision-Making in Autonomous UAV-Centric Physical AI](https://mlsanity.com/paper/2609.23695/) (2026)
- [OneWorld: Learning Consistent Physics Across Actions in World Models](https://mlsanity.com/paper/2609.30946/) (2026)
- [Learning Conditional Source Distribution via Flow Reversal for Temporal Flow Matching](https://mlsanity.com/paper/2610.05349/) (2026)

## Abstract (arXiv)

Action-conditioned world models (ACWMs) have shown strong promise for video prediction and decision-making. However, existing benchmarks are largely restricted to egocentric navigation or narrow, task-specific robotics datasets, offering only limited coverage of the rich physical interactions required for generalized world understanding. We introduce ACWM-Phys, a new benchmark for evaluating action-conditioned prediction under diverse physical dynamics in a clean, controllable simulation environment with a carefully designed action space. ACWM-Phys contains training and evaluation data spanning rigid-body dynamics, kinematics, deformable-object interactions, and particle dynamics. To evaluate both interpolation and generalization, we design in-distribution and out-of-distribution protocols with controlled shifts in interaction patterns or scene configurations. By building the benchmark in a fully controllable simulator, ACWM-Phys enables precise data collection, reproducible evaluation, and systematic analysis of model capabilities for physically grounded world modeling. Through systematic experiments on ACWM-DiT, we find that OoD generalization depends not only on the physical regime but also on effective task complexity: models generalize well on visually simple, low-dimensional interactions with clear geometric structure, but suffer larger drops on deformable contacts, high-dimensional control, and complex articulated motion. This suggests that the model still relies heavily on visual appearance patterns instead of fully learning the underlying physics. Ablations show that cross-attention improves high-dimensional action conditioning, causal VAEs outperform frame-wise encoders, and larger action spaces are harder to model but can improve generalization by providing richer control signals. These findings guide the design of physically grounded world models.