# DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control

arXiv: https://arxiv.org/abs/2603.10448 · HTML: https://mlsanity.com/paper/2603.10448/

## Key facts

- Cited by 76 papers in the map (data through 14 Oct 2026).
- Placed in Human-video and latent-action pretraining, one of 13 directions in Vision-language-action models.
- Builds on 45 of its 49 references that are in the map.
- First posted on arXiv 11 Mar 2026.

## Placed in

- Vision-language-action models / [Human-video and latent-action pretraining](https://mlsanity.com/vla/human-video-and-latent-action-pretraining/)
- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 30 of 45)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025)
- [Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success](https://mlsanity.com/paper/2502.19645/) (2025)
- [Open-Sora: Democratizing Efficient Video Production for All](https://mlsanity.com/paper/2412.20404/) (2024)
- [Learning Universal Policies via Text-Guided Video Generation](https://mlsanity.com/paper/2302.00111/) (2023)
- [UniVLA: Learning to Act Anywhere with Task-centric Latent Actions](https://mlsanity.com/paper/2505.06111/) (2025)
- [RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots](https://mlsanity.com/paper/2406.02523/) (2024)
- [WorldVLA: Towards Autoregressive Action World Model](https://mlsanity.com/paper/2506.21539/) (2025)
- [Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models](https://mlsanity.com/paper/2402.07865/) (2024)
- [Latent Action Pretraining from Videos](https://mlsanity.com/paper/2410.11758/) (2024)
- [Causal World Modeling for Robot Control](https://mlsanity.com/paper/2601.21998/) (2026)
- [π*0.6: a VLA That Learns From Experience](https://mlsanity.com/paper/2511.14759/) (2025)
- [Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning](https://mlsanity.com/paper/2601.16163/) (2026)
- [UniSim: A Neural Closed-Loop Sensor Simulator](https://mlsanity.com/paper/2308.01898/) (2023)
- [Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations](https://mlsanity.com/paper/2412.14803/) (2024)
- [Deep Visual Foresight for Planning Robot Motion](https://mlsanity.com/paper/1610.00696/) (2016)
- [Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer](https://mlsanity.com/paper/2511.22699/) (2025)

## Built on by (most cited first, 30 of 76)

- [Unified 4D World Action Modeling from Video Priors with Asynchronous Denoising](https://mlsanity.com/paper/2604.26694/) (2026)
- [World Action Models: The Next Frontier in Embodied AI](https://mlsanity.com/paper/2605.12090/) (2026)
- [World Model for Robot Learning: A Comprehensive Survey](https://mlsanity.com/paper/2605.00080/) (2026)
- [ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?](https://mlsanity.com/paper/2606.19531/) (2026)
- [Xiaomi-Robotics-1: Scaling Vision-Language-Action Models with over 100K Hours of Real-World Trajectories](https://mlsanity.com/paper/2607.15330/) (2026)
- [EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Data](https://mlsanity.com/paper/2607.08436/) (2026)
- [Faster-WAM: Efficient Inference-Time Future Conditioning for Robust World Action Models](https://mlsanity.com/paper/2608.04404/) (2026)
- [MaskWAM: Unifying Mask Prompting and Prediction for World-Action Models](https://mlsanity.com/paper/2606.13515/) (2026)
- [Learning 4D Geometric Priors for Inference-Efficient World Action Models](https://mlsanity.com/paper/2607.05468/) (2026)
- [Disentangled Robot Learning via Separate Forward and Inverse Dynamics Pretraining](https://mlsanity.com/paper/2604.16391/) (2026)
- [MemoryWAM: Efficient World Action Modeling with Persistent Memory](https://mlsanity.com/paper/2606.20562/) (2026)
- [Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization](https://mlsanity.com/paper/2608.26103/) (2026)
- [World-Language-Action Model for Unified World Modeling, Language Reasoning, and Action Synthesis](https://mlsanity.com/paper/2606.05979/) (2026)
- [World Action Models: A Survey](https://mlsanity.com/paper/2606.20781/) (2026)
- [Making Foresight Actionable: Repurposing Representation Alignment in World Action Models](https://mlsanity.com/paper/2606.12217/) (2026)
- [GigaWorld-Policy-0.5: A Faster and Stronger WAM Empowered by AutoResearch](https://mlsanity.com/paper/2607.13960/) (2026)
- [MotionWAM: Towards Foundation World Action Models for Real-Time Humanoid Loco-Manipulation](https://mlsanity.com/paper/2606.09215/) (2026)
- [Faster-WAM: Do World Action Models Need Deep Action Modules?](https://mlsanity.com/paper/2608.02365/) (2026)
- [SpatialVAM:Spatial-Aware Multi-View Video Diffusion as a Data-Efficient Robot Policy](https://mlsanity.com/paper/2604.03181/) (2026)
- [WAM-TTT: Steering World-Action Models by Watching Human Play at Test Time](https://mlsanity.com/paper/2607.06988/) (2026)
- [SANTS: A State-Adaptive Scheduler for World Action Models](https://mlsanity.com/paper/2605.27947/) (2026)
- [ω-0: A Latent Predictive World Action Model for Concurrent Humanoid Loco-Manipulation](https://mlsanity.com/paper/2608.06375/) (2026)
- [World-Action Models for Robot Learning and Control: A Survey](https://mlsanity.com/paper/2609.16074/) (2026)
- [DIM-WAM: World-Action Modeling with Diverse Historical Event Memory](https://mlsanity.com/paper/2606.27677/) (2026)
- [Colosseum V2: Benchmarking Generalization for Vision-Language-Action Models](https://mlsanity.com/paper/2605.27759/) (2026)
- [Discrete-WAM: Unified Discrete Vision-Action Token Editing for World-Policy Learning](https://mlsanity.com/paper/2606.05645/) (2026)
- [From World Models to World Action Models: A Concise Tutorial for Robotics](https://mlsanity.com/paper/2607.00836/) (2026)
- [DeVA: Decoupled Video-Action Model with physical guidance for robot policy learning](https://mlsanity.com/paper/2607.24159/) (2026)
- [Foresight Without Seeing: Latent Futures for World Action Models](https://mlsanity.com/paper/2608.11605/) (2026)
- [GIFT: Guided Intermediate Feature Training via Action-Oriented Structural Supervision for Robotic Manipulation](https://mlsanity.com/paper/2609.04193/) (2026)

## Abstract (arXiv)

Vision-Language-Action (VLA) models have emerged as a promising paradigm for robot learning, but their representations are still largely inherited from static image-text pretraining, leaving physical dynamics to be learned from comparatively limited action data. Generative video models, by contrast, encode rich spatiotemporal structure and implicit physics, making them a compelling foundation for robotic manipulation. But their potentials are not fully explored in the literature. To bridge the gap, we introduce DiT4DiT, an end-to-end Video-Action Model that couples a video Diffusion Transformer with an action Diffusion Transformer in a unified cascaded framework. Instead of relying on reconstructed future frames, DiT4DiT extracts intermediate denoising features from the video generation process and uses them as temporally grounded conditions for action prediction. We further propose a dual flow-matching objective with decoupled timesteps and noise scales for video prediction, hidden-state extraction, and action inference, enabling coherent joint training of both modules. Across simulation and real-world benchmarks, DiT4DiT achieves state-of-the-art results, reaching average success rates of 98.6% on LIBERO and 50.8% on RoboCasa GR1 while using substantially less training data. On the Unitree G1 robot, it also delivers superior real-world performance and strong zero-shot generalization. Importantly, DiT4DiT improves sample efficiency by over 10x and speeds up convergence by up to 7x, demonstrating that video generation can serve as an effective scaling proxy for robot policy learning. We release code and models at https://dit4dit.github.io/.