# WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generation

arXiv: https://arxiv.org/abs/2503.08153 · HTML: https://mlsanity.com/paper/2503.08153/

## Key facts

- Cited by 90 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 45 references that are in the map.
- First posted on arXiv 11 Mar 2025.

## Builds on (most cited first, 30 of 34)

- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [Scaling Vision with Sparse Mixture of Experts](https://mlsanity.com/paper/2106.05974/) (2021)
- [ModelScope Text-to-Video Technical Report](https://mlsanity.com/paper/2308.06571/) (2023)
- [Open-Sora: Democratizing Efficient Video Production for All](https://mlsanity.com/paper/2412.20404/) (2024)
- [VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models](https://mlsanity.com/paper/2401.09047/) (2024)
- [OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation](https://mlsanity.com/paper/2407.02371/) (2024)
- [InternVideo2: Scaling Foundation Models for Multimodal Video Understanding](https://mlsanity.com/paper/2403.15377/) (2024)
- [Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding](https://mlsanity.com/paper/2405.08748/) (2024)
- [Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning](https://mlsanity.com/paper/2311.10709/) (2023)
- [VideoPhy: Evaluating Physical Commonsense for Video Generation](https://mlsanity.com/paper/2406.03520/) (2024)
- [How Far is Video Generation from World Model: A Physical Law Perspective](https://mlsanity.com/paper/2411.02385/) (2024)
- [PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation](https://mlsanity.com/paper/2409.18964/) (2024)
- [Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation](https://mlsanity.com/paper/2410.05363/) (2024)
- [Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model](https://mlsanity.com/paper/2502.10248/) (2025)
- [Do generative video models understand physical principles?](https://mlsanity.com/paper/2501.09038/) (2025)
- [Tarsier: Recipes for Training and Evaluating Large Video Description Models](https://mlsanity.com/paper/2407.00634/) (2024)
- [Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Content](https://mlsanity.com/paper/2410.08260/) (2024)
- [Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyond](https://mlsanity.com/paper/2405.03520/) (2024)
- [Pandora: Towards General World Model with Natural Language Actions and Video States](https://mlsanity.com/paper/2406.09455/) (2024)
- [PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generation](https://mlsanity.com/paper/2412.00596/) (2024)
- [MotionCraft: Physics-based Zero-Shot Video Generation](https://mlsanity.com/paper/2405.13557/) (2024)

## Built on by (most cited first, 30 of 90)

- [VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models](https://mlsanity.com/paper/2505.23656/) (2025)
- [NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamics](https://mlsanity.com/paper/2509.21309/) (2025)
- [T2VPhysBench: A First-Principles Benchmark for Physical Consistency in Text-to-Video Generation](https://mlsanity.com/paper/2505.00337/) (2025)
- [Inference-time Physics Alignment of Video Generative Models with Latent World Models](https://mlsanity.com/paper/2601.10553/) (2026)
- ["PhyWorldBench": A Comprehensive Evaluation of Physical Realism in Text-to-Video Models](https://mlsanity.com/paper/2507.13428/) (2025)
- [PAI-Bench: A Comprehensive Benchmark For Physical AI](https://mlsanity.com/paper/2512.01989/) (2025)
- [Rethinking Video Generation Model for the Embodied World](https://mlsanity.com/paper/2601.15282/) (2026)
- [Exploring the Evolution of Physics Cognition in Video Generation: A Survey](https://mlsanity.com/paper/2503.21765/) (2025)
- [What about gravity in video generation? Post-Training Newton's Laws with Verifiable Rewards](https://mlsanity.com/paper/2512.00425/) (2025)
- [MultiWorld: Scalable Multi-Agent Multi-View Video World Models](https://mlsanity.com/paper/2604.18564/) (2026)
- [Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generation](https://mlsanity.com/paper/2508.10858/) (2025)
- [PlanGen: Towards Unified Layout Planning and Image Generation in Auto-Regressive Vision Language Models](https://mlsanity.com/paper/2503.10127/) (2025)
- [MoAlign: Motion-Centric Representation Alignment for Video Diffusion Models](https://mlsanity.com/paper/2510.19022/) (2025)
- [Simulating the Visual World with Artificial Intelligence: A Roadmap](https://mlsanity.com/paper/2511.08585/) (2025)
- [Self-Refining Video Sampling](https://mlsanity.com/paper/2601.18577/) (2026)
- [Uni-Layout: Integrating Human Feedback in Unified Layout Generation and Evaluation](https://mlsanity.com/paper/2508.02374/) (2025)
- [Physion-Eval: Evaluating Physical Realism in Generated Video via Human Reasoning](https://mlsanity.com/paper/2603.19607/) (2026)
- [Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoning](https://mlsanity.com/paper/2608.27549/) (2026)
- [Enhancing Physical Plausibility in Video Generation by Reasoning the Implausibility](https://mlsanity.com/paper/2509.24702/) (2025)
- [RULER-Bench: Probing Rule-based Reasoning Abilities of Next-level Video Generation Models for Vision Foundation Intelligence](https://mlsanity.com/paper/2512.02622/) (2025)
- [SCOPE: Simulating Cross-game Operations in Playable Environments for FPS World Models](https://mlsanity.com/paper/2605.23345/) (2026)
- [PhysMaster: Mastering Physical Representation for Video Generation via Reinforcement Learning](https://mlsanity.com/paper/2510.13809/) (2025)
- [Phantom: Physics-Infused Video Generation via Joint Modeling of Visual and Latent Physical Dynamics](https://mlsanity.com/paper/2604.08503/) (2026)
- [NEWTON: Agentic Planning for Physically Grounded Video Generation](https://mlsanity.com/paper/2605.18396/) (2026)
- [ProPhy: Progressive Physical Alignment for Dynamic World Simulation](https://mlsanity.com/paper/2512.05564/) (2025)
- [4DWorldBench: A Comprehensive Evaluation Framework for 3D/4D World Generation Models](https://mlsanity.com/paper/2511.19836/) (2025)
- [ACWM-Phys: Investigating Generalized Physical Interaction in Action-Conditioned Video World Models](https://mlsanity.com/paper/2605.08567/) (2026)
- [RDPO: Real Data Preference Optimization for Physics Consistency Video Generation](https://mlsanity.com/paper/2506.18655/) (2025)
- [PhysInOne: Visual Physics Learning and Reasoning in One Suite](https://mlsanity.com/paper/2604.09415/) (2026)
- [RAGAR: Retrieval Augmented Personalized Image Generation Guided by Recommendation](https://mlsanity.com/paper/2505.01657/) (2025)

## Abstract (arXiv)

Recent rapid advancements in text-to-video (T2V) generation, such as SoRA and Kling, have shown great potential for building world simulators. However, current T2V models struggle to grasp abstract physical principles and generate videos that adhere to physical laws. This challenge arises primarily from a lack of clear guidance on physical information due to a significant gap between abstract physical principles and generation models. To this end, we introduce the World Simulator Assistant (WISA), an effective framework for decomposing and incorporating physical principles into T2V models. Specifically, WISA decomposes physical principles into textual physical descriptions, qualitative physical categories, and quantitative physical properties. To effectively embed these physical attributes into the generation process, WISA incorporates several key designs, including Mixture-of-Physical-Experts Attention (MoPA) and a Physical Classifier, enhancing the model's physics awareness. Furthermore, most existing datasets feature videos where physical phenomena are either weakly represented or entangled with multiple co-occurring processes, limiting their suitability as dedicated resources for learning explicit physical principles. We propose a novel video dataset, WISA-32K, collected based on qualitative physical categories. It consists of 32,000 videos, representing 17 physical laws across three domains of physics: dynamics, thermodynamics, and optics. Experimental results demonstrate that WISA can effectively enhance the compatibility of T2V models with real-world physical laws, achieving a considerable improvement on the VideoPhy benchmark. The visual exhibitions of WISA and WISA-32K are available in the https://360cvgroup.github.io/WISA/.