mlsanityData to 7 Oct 2026

GenTron: Diffusion Transformers for Image and Video Generation

arXiv 2312.04557 ↗First posted 7 Dec 2023cs.CVcited by 80

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

In this study, we explore Transformer-based diffusion models for image and video generation. Despite the dominance of Transformer architectures in various fields due to their flexibility and scalability, the visual generative domain primarily utilizes CNN-based U-Net architectures, particularly in diffusion-based models. We introduce GenTron, a family of Generative models employing Transformer-based diffusion, to address this gap. Our initial step was to adapt Diffusion Transformers (DiTs) from class to text conditioning, a process involving thorough empirical exploration of the conditioning mechanism. We then scale GenTron from approximately 900M to over 3B parameters, observing significant improvements in visual quality. Furthermore, we extend GenTron to text-to-video generation, incorporating novel motion-free guidance to enhance video quality. In human evaluations against SDXL, GenTron achieves a 51.1% win rate in visual quality (with a 19.8% draw rate), and a 42.3% win rate in text alignment (with a 42.9% draw rate). GenTron also excels in the T2I-CompBench, underscoring its strengths in compositional generation. We believe this work will provide meaningful insights and serve as a valuable reference for future research.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 58 of its 71 references that are in the map. First posted on arXiv 7 Dec 2023.

Builds on

58 of 71 references.

Closest ancestors · 12

Same direction, newest first

  1. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis30 Sep
  2. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis4 Jul
  3. 2023Fast Training of Diffusion Models with Masked Transformers15 Jun
  4. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer25 Mar
  5. 2022Scalable Diffusion Models with Transformers19 Dec
  6. 2022All are Worth Words: A ViT Backbone for Diffusion Models25 Sep
  7. 2022Classifier-Free Diffusion Guidance26 Jul
  8. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2 Jun
  9. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  10. 2021Cascaded Diffusion Models for High Fidelity Image Generation30 May
  11. 2020Denoising Diffusion Implicit Models6 Oct
  12. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
34 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  3. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  7. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  8. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  9. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  10. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  11. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  12. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  13. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  14. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  15. 2021Scaling Vision Transformerscited by 993
  16. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  17. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  18. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  19. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  20. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  21. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  22. 2023ModelScope Text-to-Video Technical Reportcited by 651
  23. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  24. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  25. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  26. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 462
  27. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  28. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 330
  29. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 278
  30. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 253
  31. 2023FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editingcited by 165
  32. 2023RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Pathscited by 141
  33. 2023Dolfin: Diffusion Layout Transformers without Autoencodercited by 23
  34. 2019Jointly Trained Image and Video Generation using Residual Vectorscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 24; 2025: 34; 2026: 22.

80 papers.

Sort:

By area:

  1. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  2. 2024PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generationcited by 326
  3. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  4. 2024Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Contentcited by 132
  5. 2025PixelFlow: Pixel-Space Generative Models with Flowcited by 91
  6. 2024ZigMa: A DiT-style Zigzag Mamba Diffusion Modelcited by 84
  7. 2025What matters for Representation Alignment: Global Information or Spatial Structure?cited by 76
  8. 2024Allegro: Open the Black Box of Commercial-Level Video Generation Modelcited by 68
  9. 2025Goku: Flow Based Video Generative Foundation Modelscited by 67
  10. 2025STAR: Spatial-Temporal Augmentation with Text-to-Video Models for Real-World Video Super-Resolutioncited by 56
  11. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
  12. 2024MarDini: Masked Autoregressive Diffusion for Video Generation at Scalecited by 49
38 more
  1. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 42
  2. 2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practicescited by 39
  3. 2024Mind the Time: Temporally-Controlled Multi-Event Video Generationcited by 39
  4. 2024Video Motion Transfer with Diffusion Transformerscited by 39
  5. 2025ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Modelscited by 37
  6. 2024IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generationcited by 32
  7. 2024DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factorscited by 31
  8. 2025OmniHuman-1.5: Instilling an Active Mind in Avatars via Cognitive Simulationcited by 30
  9. 2025Movie Weaver: Tuning-Free Multi-Concept Video Personalization with Anchored Promptscited by 28
  10. 2024From Sora What We Can See: A Survey of Text-to-Video Generationcited by 26
  11. 2025AR-GRPO: Training Autoregressive Image Generation Models via Reinforcement Learningcited by 22
  12. 2024LaVin-DiT: Large Vision Diffusion Transformercited by 21
  13. 2025Macro-from-Micro Planning for High-Quality and Parallelized Autoregressive Long Video Generationcited by 19
  14. 2025TR-DQ: Time-Rotation Diffusion Quantizationcited by 18
  15. 2024Kubrick: Multimodal Agent Collaborations for Synthetic Video Generationcited by 16
  16. 2024VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generationcited by 16
  17. 2025Simulating the Real World: A Unified Survey of Multimodal Generative Modelscited by 14
  18. 2024Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulationcited by 13
  19. 2026Robotic Video World Models: A Survey of Applications, Research Challenges, Future Directionscited by 12
  20. 2024SPAgent: Adaptive Task Decomposition and Model Selection for General Video Generation and Editingcited by 11
  21. 2025Can Video Diffusion Model Reconstruct 4D Geometry?cited by 10
  22. 2025Image Can Bring Your Memory Back: A Novel Multi-Modal Guided Attack against Image Generation Model Unlearningcited by 9
  23. 2025Scaling Zero-Shot Reference-to-Video Generationcited by 9
  24. 2024Multi-modal Generative AI: Multi-modal LLMs, Diffusions, and the Unificationcited by 8
  25. 2025Scaling Sequence-to-Sequence Generative Neural Renderingcited by 8
  26. 2025HiStream: Efficient High-Resolution Video Generation via Redundancy-Eliminated Streamingcited by 8
  27. 2025DynamiCtrl: Rethinking the Basic Structure and the Role of Text for High-quality Human Image Animationcited by 7
  28. 2024Trustworthy Text-to-Image Diffusion Models: A Timely and Focused Surveycited by 6
  29. 2026MultiAnimate: Pose-Guided Image Animation Made Extensiblecited by 6
  30. 2026Evolution of Video Generative Foundationscited by 6
  31. 2025Sampling 3D Molecular Conformers with Diffusion Transformerscited by 5
  32. 2025Coherent Audio-Visual Editing via Conditional Audio Generation Following Video Editscited by 5
  33. 2026AdaCluster: Adaptive Query-Key Clustering for Sparse Attention in Video Generationcited by 5
  34. 2024FlexCache: Flexible Approximate Cache System for Video Diffusioncited by 4
  35. 2025Boosting Resolution Generalization of Diffusion Transformers with Randomized Positional Encodingscited by 4
  36. 2025Flow Autoencoders are Effective Protein Tokenizerscited by 4
  37. 2025LDGen: Enhancing Text-to-Image Synthesis via Large Language Model-Driven Language Representationcited by 3
  38. 2025Seeing Clearly, Forgetting Deeply: Revisiting Fine-Tuned Video Generators for Driving Simulationcited by 3

50 most cited of 80 All 80 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Text Conditioning: A Systematic Study of MLLM-DiT Fusion for Video Generation
  2. 2025DiT-Air: Revisiting the Efficiency of Diffusion Model Architecture Design in Text to Image Generation
  3. 2025DyDiT++: Diffusion Transformers with Timestep and Spatial Dynamics for Efficient Visual Generation
  4. 2024SmoothCache: A Universal Inference Acceleration Technique for Diffusion Transformers
  5. 2026Calibri: Enhancing Diffusion Transformers via Parameter-Efficient Calibration
  6. 2026DiffusionBench: On Holistic Evaluation of Diffusion Transformers
  7. 2023State of the Art on Diffusion Models for Visual Computing
  8. 2023Photorealistic Video Generation with Diffusion Models