# DiVE: DiT-based Video Generation with Enhanced Control

arXiv: https://arxiv.org/abs/2409.01595 · HTML: https://mlsanity.com/paper/2409.01595/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 15 references that are in the map.
- First posted on arXiv 3 Sep 2024.

## Builds on (most cited first, 15 of 15)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [MUSIQ: Multi-scale Image Quality Transformer](https://mlsanity.com/paper/2108.05997/) (2021)
- [Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection](https://mlsanity.com/paper/2303.11926/) (2023)
- [MagicDrive: Street View Generation with Diverse 3D Geometry Control](https://mlsanity.com/paper/2310.02601/) (2023)
- [PIXART-δ: Fast and Controllable Image Generation with Latent Consistency Models](https://mlsanity.com/paper/2401.05252/) (2024)
- [Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generation](https://mlsanity.com/paper/2406.01349/) (2024)

## Built on by (most cited first, 30 of 31)

- [Controllable Video Generation: A Survey](https://mlsanity.com/paper/2507.16869/) (2025)
- [MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Control](https://mlsanity.com/paper/2411.13807/) (2024)
- [3D and 4D World Modeling: A Survey](https://mlsanity.com/paper/2509.07996/) (2025)
- [DreamForge: Motion-Aware Autoregressive Video Generation for Multi-View Driving Scenes](https://mlsanity.com/paper/2409.04003/) (2024)
- [MaskGWM: A Generalizable Driving World Model with Video Mask Reconstruction](https://mlsanity.com/paper/2502.11663/) (2025)
- [UniMLVG: Unified Framework for Multi-view Long Video Generation with Comprehensive Control Capabilities for Autonomous Driving](https://mlsanity.com/paper/2412.04842/) (2024)
- [WorldLens: Full-Spectrum Evaluations of Driving World Models in Real World](https://mlsanity.com/paper/2512.10958/) (2025)
- [UniFuture: A 4D Driving World Model for Future Generation and Perception](https://mlsanity.com/paper/2503.13587/) (2025)
- [Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistency](https://mlsanity.com/paper/2506.07497/) (2025)
- [Simulating the Visual World with Artificial Intelligence: A Roadmap](https://mlsanity.com/paper/2511.08585/) (2025)
- [Rethinking Driving World Model as Synthetic Data Generator for Perception Tasks](https://mlsanity.com/paper/2510.19195/) (2025)
- [WorldSplat: Gaussian-Centric Feed-Forward 4D Scene Generation for Autonomous Driving](https://mlsanity.com/paper/2509.23402/) (2025)
- [CoGen: 3D Consistent Video Generation via Adaptive Conditioning for Autonomous Driving](https://mlsanity.com/paper/2503.22231/) (2025)
- [Any2Caption:Interpreting Any Condition to Caption for Controllable Video Generation](https://mlsanity.com/paper/2503.24379/) (2025)
- [DiVE: Efficient Multi-View Driving Scenes Generation Based on Video Diffusion Transformer](https://mlsanity.com/paper/2504.19614/) (2025)
- [LongDWM: Cross-Granularity Distillation for Building a Long-Term Driving World Model](https://mlsanity.com/paper/2506.01546/) (2025)
- [OccTENS: 3D Occupancy World Model via Temporal Next-Scale Prediction](https://mlsanity.com/paper/2509.03887/) (2025)
- [PosePilot: Steering Camera Pose for Generative World Models with Self-supervised Depth](https://mlsanity.com/paper/2505.01729/) (2025)
- [OneVAE: Joint Discrete and Continuous Optimization Helps Discrete Video VAE Train Better](https://mlsanity.com/paper/2508.09857/) (2025)
- [FAR-Drive: Frame-AutoRegressive Video Generation in Closed-Loop Autonomous Driving](https://mlsanity.com/paper/2603.14938/) (2026)
- [CorrectAD: A Self-Correcting Agentic System to Improve End-to-end Planning in Autonomous Driving](https://mlsanity.com/paper/2511.13297/) (2025)
- [OmniDrive: An LLM-Choreographed Multi-Agent World Model with Unified Latent Co-Compression for Multi-View Driving Video Generation](https://mlsanity.com/paper/2606.17536/) (2026)
- [ParaScale: Scale-Calibrated Camera-Motion Transfer via a Gauge-Invariant Parallax Number](https://mlsanity.com/paper/2606.19805/) (2026)
- [CVD-STORM: Cross-View Video Diffusion with Spatial-Temporal Reconstruction Model for Autonomous Driving](https://mlsanity.com/paper/2510.07944/) (2025)
- [Autoregressive Video Autoencoder with Decoupled Temporal and Spatial Context](https://mlsanity.com/paper/2512.11293/) (2025)
- [OrthoMotion:Disentangling Camera and Subject Motion via Geometry Semantics Orthogonal Attention](https://mlsanity.com/paper/2606.22835/) (2026)
- [MSEditor: Toward Consistent Multi-Shot Video Editing](https://mlsanity.com/paper/2608.17559/) (2026)
- [MM-MovieDubber: Towards Multi-Modal Learning for Multi-Modal Movie Dubbing](https://mlsanity.com/paper/2505.16279/) (2025)
- [LSA: Localized Semantic Alignment for Enhancing Temporal Consistency in Traffic Video Generation](https://mlsanity.com/paper/2602.05966/) (2026)
- [HelloWorld: Towards Practical Applications of Generative Driving World Models](https://mlsanity.com/paper/2609.28931/) (2026)

## Abstract (arXiv)

Generating high-fidelity, temporally consistent videos in autonomous driving scenarios faces a significant challenge, e.g. problematic maneuvers in corner cases. Despite recent video generation works are proposed to tackcle the mentioned problem, i.e. models built on top of Diffusion Transformers (DiT), works are still missing which are targeted on exploring the potential for multi-view videos generation scenarios. Noticeably, we propose the first DiT-based framework specifically designed for generating temporally and multi-view consistent videos which precisely match the given bird's-eye view layouts control. Specifically, the proposed framework leverages a parameter-free spatial view-inflated attention mechanism to guarantee the cross-view consistency, where joint cross-attention modules and ControlNet-Transformer are integrated to further improve the precision of control. To demonstrate our advantages, we extensively investigate the qualitative comparisons on nuScenes dataset, particularly in some most challenging corner cases. In summary, the effectiveness of our proposed method in producing long, controllable, and highly consistent videos under difficult conditions is proven to be effective.