# DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation

arXiv: https://arxiv.org/abs/2412.18597 · HTML: https://mlsanity.com/paper/2412.18597/

## Key facts

- Cited by 74 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 46 of its 49 references that are in the map.
- First posted on arXiv 24 Dec 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 46)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [WorldSimBench: Towards Video Generation Models as World Simulators](https://mlsanity.com/paper/2410.18072/) (2024)
- [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023)
- [ModelScope Text-to-Video Technical Report](https://mlsanity.com/paper/2308.06571/) (2023)
- [Movie Gen: A Cast of Media Foundation Models](https://mlsanity.com/paper/2410.13720/) (2024)
- [VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models](https://mlsanity.com/paper/2401.09047/) (2024)
- [MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation](https://mlsanity.com/paper/2302.08113/) (2023)
- [VideoCrafter1: Open Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2310.19512/) (2023)
- [Zero-shot Image-to-Image Translation](https://mlsanity.com/paper/2302.03027/) (2023)
- [Phenaki: Variable Length Video Generation From Open Domain Textual Description](https://mlsanity.com/paper/2210.02399/) (2022)
- [FateZero: Fusing Attentions for Zero-shot Text-based Video Editing](https://mlsanity.com/paper/2303.09535/) (2023)
- [MagicVideo: Efficient Video Generation With Latent Diffusion Models](https://mlsanity.com/paper/2211.11018/) (2022)
- [Latent Video Diffusion Models for High-Fidelity Long Video Generation](https://mlsanity.com/paper/2211.13221/) (2022)
- [LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models](https://mlsanity.com/paper/2309.15103/) (2023)
- [EvalCrafter: Benchmarking and Evaluating Large Video Generation Models](https://mlsanity.com/paper/2310.11440/) (2023)

## Built on by (most cited first, 30 of 74)

- [Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Models](https://mlsanity.com/paper/2504.12626/) (2025)
- [Long Context Tuning for Video Generation](https://mlsanity.com/paper/2503.10589/) (2025)
- [VideoPainter: Any-length Video Inpainting and Editing with Plug-and-Play Context Control](https://mlsanity.com/paper/2503.05639/) (2025)
- [OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation](https://mlsanity.com/paper/2505.20292/) (2025)
- [MOVA: Towards Scalable and Synchronized Video-Audio Generation](https://mlsanity.com/paper/2602.08794/) (2026)
- [Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Control](https://mlsanity.com/paper/2506.01943/) (2025)
- [Emergent Temporal Correspondences from Video Diffusion Transformers](https://mlsanity.com/paper/2506.17220/) (2025)
- [CineTrans: Learning to Generate Videos with Cinematic Transitions via Masked Diffusion Models](https://mlsanity.com/paper/2508.11484/) (2025)
- [DreamRelation: Relation-Centric Video Customization](https://mlsanity.com/paper/2503.07602/) (2025)
- [LongVie: Multimodal-Guided Controllable Ultra-Long Video Generation](https://mlsanity.com/paper/2508.03694/) (2025)
- [FreeLong++: Training-Free Long Video Generation via Multi-band SpectralFusion](https://mlsanity.com/paper/2507.00162/) (2025)
- [T2VUnlearning: A Concept Erasing Method for Text-to-Video Diffusion Models](https://mlsanity.com/paper/2505.17550/) (2025)
- [Audio-Sync Video Generation with Multi-Stream Temporal Control](https://mlsanity.com/paper/2506.08003/) (2025)
- [Physical Simulator In-the-Loop Video Generation](https://mlsanity.com/paper/2603.06408/) (2026)
- [ScalingNoise: Scaling Inference-Time Search for Generating Infinite Videos](https://mlsanity.com/paper/2503.16400/) (2025)
- [UltraViCo: Breaking Extrapolation Limits in Video Diffusion Transformers](https://mlsanity.com/paper/2511.20123/) (2025)
- [EditMGT: Unleashing Potentials of Masked Generative Transformers in Image Editing](https://mlsanity.com/paper/2512.11715/) (2025)
- [DreamVE: Unified Instruction-based Image and Video Editing](https://mlsanity.com/paper/2508.06080/) (2025)
- [FilmWeaver: Weaving Consistent Multi-Shot Videos with Cache-Guided Autoregressive Diffusion](https://mlsanity.com/paper/2512.11274/) (2025)
- [PoseGen: In-Context LoRA Finetuning for Pose-Controllable Long Human Video Generation](https://mlsanity.com/paper/2508.05091/) (2025)
- [PAI-Studio: Cinematic Video Background Replacement with Camera-Aware Motion](https://mlsanity.com/paper/2606.01399/) (2026)
- [DynamiCtrl: Rethinking the Basic Structure and the Role of Text for High-quality Human Image Animation](https://mlsanity.com/paper/2503.21246/) (2025)
- [LoCoT2V-Bench: Benchmarking Long-Form and Complex Text-to-Video Generation](https://mlsanity.com/paper/2510.26412/) (2025)
- [Tuning-Free Multi-Event Long Video Generation via Synchronized Coupled Sampling](https://mlsanity.com/paper/2503.08605/) (2025)
- [GenHSI: Controllable Generation of Human-Scene Interaction Videos](https://mlsanity.com/paper/2506.19840/) (2025)
- [LongAnimation: Long Animation Generation with Dynamic Global-Local Memory](https://mlsanity.com/paper/2507.01945/) (2025)
- [ConsistEdit: Highly Consistent and Precise Training-free Visual Editing](https://mlsanity.com/paper/2510.17803/) (2025)
- [ProEdit: Inversion-based Editing From Prompts Done Right](https://mlsanity.com/paper/2512.22118/) (2025)
- [SceneTransporter: Optimal Transport-Guided Compositional Latent Diffusion for Single-Image Structured 3D Scene Generation](https://mlsanity.com/paper/2602.22785/) (2026)
- [PhysVid: Physics Aware Local Conditioning for Generative Video Models](https://mlsanity.com/paper/2603.26285/) (2026)

## Abstract (arXiv)

Sora-like video generation models have achieved remarkable progress with a Multi-Modal Diffusion Transformer MM-DiT architecture. However, the current video generation models predominantly focus on single-prompt, struggling to generate coherent scenes with multiple sequential prompts that better reflect real-world dynamic scenarios. While some pioneering works have explored multi-prompt video generation, they face significant challenges including strict training data requirements, weak prompt following, and unnatural transitions. To address these problems, we propose DiTCtrl, a training-free multi-prompt video generation method under MM-DiT architectures for the first time. Our key idea is to take the multi-prompt video generation task as temporal video editing with smooth transitions. To achieve this goal, we first analyze MM-DiT's attention mechanism, finding that the 3D full attention behaves similarly to that of the cross/self-attention blocks in the UNet-like diffusion models, enabling mask-guided precise semantic control across different prompts with attention sharing for multi-prompt video generation. Based on our careful design, the video generated by DiTCtrl achieves smooth transitions and consistent object motion given multiple sequential prompts without additional training. Besides, we also present MPVBench, a new benchmark specially designed for multi-prompt video generation to evaluate the performance of multi-prompt generation. Extensive experiments demonstrate that our method achieves state-of-the-art performance without additional training.