mlsanityData to 7 Oct 2026

Mind the Time: Temporally-Controlled Multi-Event Video Generation

arXiv 2412.05263 ↗First posted 6 Dec 2024cs.CVcited by 39

Not on the map. Why?

Abstract (arXiv)

Real-world videos consist of sequences of events. Generating such sequences with precise temporal control is infeasible with existing video generators that rely on a single paragraph of text as input. When tasked with generating multiple events described using a single prompt, such methods often ignore some of the events or fail to arrange them in the correct order. To address this limitation, we present MinT, a multi-event video generator with temporal control. Our key insight is to bind each event to a specific period in the generated video, which allows the model to focus on one event at a time. To enable time-aware interactions between event captions and video tokens, we design a time-based positional encoding method, dubbed ReRoPE. This encoding helps to guide the cross-attention operation. By fine-tuning a pre-trained video diffusion transformer on temporally grounded data, our approach produces coherent videos with smoothly connected events. For the first time in the literature, our model offers control over the timing of events in generated videos. Extensive experiments demonstrate that MinT outperforms existing commercial and open-source models by a large margin.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 97 of its 110 references that are in the map. First posted on arXiv 6 Dec 2024.

Builds on

97 of 110 references.

References · 97

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2023GPT-4 Technical Reportcited by 18,132
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2020Denoising Diffusion Implicit Modelscited by 9,655
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  11. 2022Scalable Diffusion Models with Transformerscited by 5,931
  12. 2022Classifier-Free Diffusion Guidancecited by 5,717
38 more
  1. 2016Layer Normalizationcited by 5,579
  2. 2022Flow Matching for Generative Modelingcited by 5,426
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  5. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  8. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  9. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  10. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  11. 2022Video Diffusion Modelscited by 2,280
  12. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  13. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  14. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  15. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  16. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  17. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  18. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  19. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  20. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  21. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  22. 2017Dense-Captioning Events in Videoscited by 789
  23. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  24. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  25. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  26. 2017Localizing Moments in Video with Natural Languagecited by 660
  27. 2023ModelScope Text-to-Video Technical Reportcited by 651
  28. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  29. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  30. 2024Movie Gen: A Cast of Media Foundation Modelscited by 614
  31. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  32. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 568
  33. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  34. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  35. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  36. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  37. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  38. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463

50 most cited of 97 All 97 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 22; 2026: 16.

39 papers.

Sort:

By area:

  1. 2025Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Modelscited by 126
  2. 2025Long Context Tuning for Video Generationcited by 110
  3. 2025DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Modelscited by 32
  4. 2025MoCha: Towards Movie-Grade Talking Character Synthesiscited by 31
  5. 2025OneStory: Coherent Multi-Shot Video Generation with Adaptive Memorycited by 31
  6. 2024Is Your World Simulator a Good Story Presenter? A Consecutive Events-Based Benchmark for Future Long Video Generationcited by 28
  7. 2025EchoShot: Multi-Shot Portrait Video Generationcited by 28
  8. 2025LoViC: Efficient Long Video Generation with Context Compressioncited by 19
  9. 2025Generative AI for Cel-Animation: A Surveycited by 17
  10. 2025AniMaker: Multi-Agent Animated Storytelling with MCTS-Driven Clip Generationcited by 17
  11. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applicationscited by 14
  12. 2025EasyV2V: A High-quality Instruction-based Video Editing Frameworkcited by 11
27 more
  1. 2025STORYANCHORS: Generating Consistent Multi-Scene Story Frames for Long-Form Narrativescited by 9
  2. 2025Tuning-Free Multi-Event Long Video Generation via Synchronized Coupled Samplingcited by 6
  3. 2025GenHSI: Controllable Generation of Human-Scene Interaction Videoscited by 6
  4. 2025A Survey on Long-Video Storytelling Generation: Architectures, Consistency, and Cinematic Qualitycited by 6
  5. 2026Prompt Relay: Inference-Time Temporal Control for Multi-Event Video Generationcited by 6
  6. 2025Flow Straight and Fast in Hilbert Space: Functional Rectified Flowcited by 5
  7. 2025TempoControl: Temporal Attention Guidance for Text-to-Video Modelscited by 5
  8. 2025VHOI: Controllable Video Generation of Human-Object Interactions from Sparse Trajectories via Motion Densificationcited by 4
  9. 2026SwitchCraft: Training-Free Multi-Event Video Generation with Attention Controlscited by 4
  10. 2026ActionParty: Multi-Subject Action Binding in Generative Video Gamescited by 4
  11. 2026TS-Attn: Temporal-wise Separable Attention for Multi-Event Video Generationcited by 4
  12. 2025A Survey: Spatiotemporal Consistency in Video Generationcited by 3
  13. 2025AlcheMinT: Fine-grained Temporal Control for Multi-Reference Consistent Video Generationcited by 3
  14. 2026iTryOn: Mastering Interactive Video Virtual Try-On with Spatial-Semantic Guidancecited by 3
  15. 2026Narrative Weaver: Towards Controllable Long-Range Visual Consistency with Multi-Modal Conditioningcited by 2
  16. 2026TIE: Time Interval Encoding for Video Generation over Eventscited by 2
  17. 2025Arbitrary Generative Video Interpolationcited by 1
  18. 2025When and Where do Events Switch in Multi-Event Video Generation?cited by 1
  19. 2026Anchoring and Rescaling Attention for Semantically Coherent Inbetweeningcited by 1
  20. 2026Delta Forcing: Trust Region Steering for Interactive Autoregressive Video Generationcited by 1
  21. 2026TunerDiT: Training-free Progressive Steering of Diffusion Transformer for Multi-Event Video Generationcited by 1
  22. 2026Crayotter: Traceable Multi-Agent Workflows for Long-Form Video Editingcited by 1
  23. 2026Compositional Video Generation via Inference-Time Guidancecited by 0
  24. 2026CineOrchestra: Unified Entity-Centric Conditioning for Cinematic Video Generationcited by 0
  25. 2026MAVIN: Multi-Shot Audio-Visual Generation with Customized Narrative Controlcited by 0
  26. 2026TPD: Temporal Prior Decoupling for Text-to-Video Diffusion Modelscited by 0
  27. 2026The Missing Temporal Link: Temporal Context Routing for Script-Driven Audio-Video Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VideoExpert: Augmented LLM for Temporal-Sensitive Video Understanding
  2. 2024Video Diffusion Transformers are In-Context Learners
  3. 2025TimeExpert: An Expert-Guided Video LLM for Video Temporal Grounding
  4. 2024DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation
  5. 2025Tempo-R0: A Video-MLLM for Temporal Video Grounding through Efficient Temporal Sensing Reinforcement Learning
  6. 2026ViKey: Enhancing Temporal Understanding in Videos via Visual Prompting
  7. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning
  8. 2026Event-T2M: Event-level Conditioning for Complex Text-to-Motion Synthesis