mlsanityData to 14 Oct 2026

FullDiT: Multi-Task Video Generative Foundation Model with Full Attention

arXiv 2503.19907 ↗First posted 25 Mar 2025cs.CVcited by 47

Not on the map. Why?

Abstract (arXiv)

Current video generative foundation models primarily focus on text-to-video tasks, providing limited control for fine-grained video content creation. Although adapter-based approaches (e.g., ControlNet) enable additional controls with minimal fine-tuning, they encounter challenges when integrating multiple conditions, including: branch conflicts between independently trained adapters, parameter redundancy leading to increased computational cost, and suboptimal performance compared to full fine-tuning. To address these challenges, we introduce FullDiT, a unified foundation model for video generation that seamlessly integrates multiple conditions via unified full-attention mechanisms. By fusing multi-task conditions into a unified sequence representation and leveraging the long-context learning ability of full self-attention to capture condition dynamics, FullDiT reduces parameter overhead, avoids conditions conflict, and shows scalability and emergent ability. We further introduce FullBench for multi-task video generation evaluation. Experiments demonstrate that FullDiT achieves state-of-the-art results, highlighting the efficacy of full-attention in complex multi-task video generation.

Key facts. Cited by 47 papers in the map (data through 14 Oct 2026). Builds on 67 of its 70 references that are in the map. First posted on arXiv 25 Mar 2025.

Builds on

67 of 70 references.

References · 67

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  3. 2022Scalable Diffusion Models with Transformerscited by 6,054
  4. 2022Flow Matching for Generative Modelingcited by 5,702
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  6. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  8. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  9. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  10. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  11. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
  12. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
38 more
  1. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,360
  2. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,351
  3. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  4. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  5. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  6. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 688
  7. 2024Movie Gen: A Cast of Media Foundation Modelscited by 644
  8. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 580
  9. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 579
  10. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  11. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  12. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  13. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 460
  14. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  15. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  16. 2024OmniGen: Unified Image Generationcited by 394
  17. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 393
  18. 2016Generating Videos with Scene Dynamicscited by 376
  19. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  20. 2023ControlVideo: Training-free Controllable Text-to-Video Generationcited by 327
  21. 2023Photorealistic Video Generation with Diffusion Modelscited by 306
  22. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 301
  23. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 296
  24. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 279
  25. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 255
  26. 2023OmniControl: Control Any Joint at Any Time for Human Motion Generationcited by 238
  27. 2024Global Structure-from-Motion Revisitedcited by 218
  28. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 188
  29. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 180
  30. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 178
  31. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 175
  32. 2024Identity-Preserving Text-to-Video Generation by Frequency Decompositioncited by 158
  33. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  34. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 150
  35. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 146
  36. 2025OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Modelscited by 129
  37. 2024ID-Animator: Zero-Shot Identity-Preserving Human Video Generationcited by 124
  38. 2024CamI2V: Camera-Controlled Image-to-Video Diffusion Modelcited by 120

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 24; 2026: 23.

47 papers.

Sort:

By area:

  1. 2025Controllable Video Generation: A Surveycited by 98
  2. 2025UniVideo: Unified Understanding, Generation, and Editing for Videoscited by 87
  3. 2025EditVerse: Unifying Image and Video Editing and Generation with In-Context Learningcited by 54
  4. 2025Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillationcited by 41
  5. 2026ShotStream: Streaming Multi-Shot Video Generation for Interactive Storytellingcited by 28
  6. 2026VINO: A Unified Visual Generator with Interleaved OmniModal Contextcited by 26
  7. 2025FullDiT2: Efficient In-Context Conditioning for Video Diffusion Transformerscited by 18
  8. 2025Phantom-Data : Towards a General Subject-Consistent Video Generation Datasetcited by 17
  9. 2025OmniVCus: Feedforward Subject-driven Video Customization with Multimodal Control Conditionscited by 17
  10. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  11. 2025Video-As-Prompt: Unified Semantic Control for Video Generationcited by 14
  12. 2025EasyV2V: A High-quality Instruction-based Video Editing Frameworkcited by 13
35 more
  1. 2026Lance: Unified Multimodal Modeling by Multi-Task Synergycited by 12
  2. 2025Any2Caption:Interpreting Any Condition to Caption for Controllable Video Generationcited by 10
  3. 2025Frame In-N-Out: Unbounded Controllable Image-to-Video Generationcited by 10
  4. 2025GenCompositor: Generative Video Compositing with Diffusion Transformercited by 9
  5. 2025LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillationcited by 9
  6. 2026Plenoptic Video Generationcited by 9
  7. 2025VideoCoF: Unified Video Editing with Temporal Reasonercited by 8
  8. 2025UnityVideo: Unified Multi-Modal Multi-Task Learning for Enhancing World-Aware Video Generationcited by 7
  9. 2026CineScene: Implicit 3D as Effective Scene Representation for Cinematic Video Generationcited by 7
  10. 2026Evolution of Video Generative Foundationscited by 7
  11. 2026MoCha:End-to-End Video Character Replacement without Structural Guidancecited by 6
  12. 2026Omni-3DEdit: Generalized Versatile 3D Editing in One-Passcited by 6
  13. 2025VideoCanvas: Unified Video Completion from Arbitrary Spatiotemporal Patches via In-Context Conditioningcited by 5
  14. 2025CETCAM: Camera-Controllable Video Generation via Consistent and Extensible Tokenizationcited by 5
  15. 2026TeleStyle: Content-Preserving Style Transfer in Images and Videoscited by 5
  16. 2026DreamVideo-Omni: Omni-Motion Controlled Multi-Subject Video Customization with Latent Identity Reinforcement Learningcited by 4
  17. 2026Gloria: Consistent Character Video Generation via Content Anchorscited by 3
  18. 2026MMPhysVideo: Physically Plausible Video Generation Through Joint RGB-Perception Modelingcited by 3
  19. 2026SWIFT: Prompt-Adaptive Memory for Efficient Interactive Long Video Generationcited by 3
  20. 2026Smart-Insertion-V: Photorealistic Video Insertion via a Closed-Loop Feedback Dual-Stream Frameworkcited by 3
  21. 2026AnchorWorld: Embodied Egocentric World Simulation with View-based Evolution Customizationcited by 3
  22. 2026Composing Driving Worlds through Disentangled Control for Adversarial Scenario Generationcited by 2
  23. 2026Lighting-grounded Video Generation with Renderer-based Agent Reasoningcited by 2
  24. 2026How Far Are Video Models from True Multimodal Reasoning?cited by 2
  25. 2025UniMMVSR: A Unified Multi-Modal Framework for Cascaded Video Super-Resolutioncited by 1
  26. 2025Can Image-To-Video Models Simulate Pedestrian Dynamics?cited by 1
  27. 2025CustomX: Unified Character, Action, and Scene Customization in Video World Modelscited by 1
  28. 2026MoZoo:Unleashing Video Diffusion power in animal fur and muscle simulationcited by 1
  29. 2025PEDRA: Evaluating the Realism of Pedestrian Dynamics in Video Generationcited by 0
  30. 2025A Reason-then-Describe Instruction Interpreter for Controllable Video Generationcited by 0
  31. 2025In-Context Audio Control of Video Diffusion Transformerscited by 0
  32. 2026Geometry-Instructed Video Editingcited by 0
  33. 2026ICDepth: Taming Video Diffusion Models for Video Depth Estimation via In-Context Conditioningcited by 0
  34. 2026RefVideo-6M: A Reliable Reference-Based Dataset for Instructional Video Editingcited by 0
  35. 2026CatPortrait: Taming Video Generative Models for Portrait Animation via In-context Conditioningcited by 0