mlsanityData to 7 Oct 2026

N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtion

arXiv 2111.12417 ↗First posted 24 Nov 2021cs.CV cs.AIcited by 253

Not on the map. Why?

Cited by 81 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper presents a unified multimodal pre-trained model called N"UWA that can generate new or manipulate existing visual data (i.e., images and videos) for various visual synthesis tasks. To cover language, image, and video at the same time for different scenarios, a 3D transformer encoder-decoder framework is designed, which can not only deal with videos as 3D data but also adapt to texts and images as 1D and 2D data, respectively. A 3D Nearby Attention (3DNA) mechanism is also proposed to consider the nature of the visual data and reduce the computational complexity. We evaluate N"UWA on 8 downstream tasks. Compared to several strong baselines, N"UWA achieves state-of-the-art results on text-to-image generation, text-to-video generation, video prediction, etc. Furthermore, it also shows surprisingly good zero-shot capabilities on text-guided image and video manipulation tasks. Project repo is https://github.com/microsoft/NUWA.

Key facts. Cited by 253 papers in the map (data through 7 Oct 2026). Builds on 45 of its 52 references that are in the map. First posted on arXiv 24 Nov 2021.

Builds on

45 of 52 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2017Neural Discrete Representation Learningcited by 4,650
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  9. 2016Improved Techniques for Training GANscited by 2,938
  10. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  11. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  12. 2017The Kinetics Human Action Video Datasetcited by 1,894
33 more
  1. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  2. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  3. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  4. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  5. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  6. 2016Pixel Recurrent Neural Networkscited by 814
  7. 2018Image Transformercited by 770
  8. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  9. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  10. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  11. 2019Segmentation Transformer: Object-Contextual Representations for Semantic Segmentationcited by 573
  12. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  13. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  14. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  15. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  16. 2019Axial Attention in Multidimensional Transformerscited by 350
  17. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  18. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  19. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
  20. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  21. 2018Stochastic Video Generation with a Learned Priorcited by 244
  22. 2017Stochastic Variational Video Predictioncited by 234
  23. 2018Moments in Time Dataset: one million videos for event understandingcited by 215
  24. 2019Scaling Autoregressive Video Modelscited by 189
  25. 2018Stochastic Adversarial Video Predictioncited by 173
  26. 2017Video Generation From Textcited by 153
  27. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 137
  28. 2021Paint by Wordcited by 107
  29. 2020Stochastic Latent Residual Video Predictioncited by 106
  30. 2020Latent Video Transformercited by 106
  31. 2019VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generationcited by 74
  32. 2021CCVS: Context-aware Controllable Video Synthesiscited by 68
  33. 2020Transformation-based Adversarial Video Prediction on Large-Scale Datacited by 66

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 43; 2023: 108; 2024: 63; 2025: 28; 2026: 9.

253 papers.

Sort:

By area:

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  2. 2022Video Diffusion Modelscited by 2,280
  3. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  4. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  5. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  6. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  7. 2025Qwen-Image Technical Reportcited by 973
  8. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  9. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  10. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  11. 2024Genie: Generative Interactive Environmentscited by 706
  12. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
38 more
  1. 2023ModelScope Text-to-Video Technical Reportcited by 651
  2. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  3. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  4. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  5. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  6. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  7. 2023MusicLM: Generating Music From Textcited by 442
  8. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  9. 2022MAGVIT: Masked Generative Video Transformercited by 341
  10. 2023ControlVideo: Training-free Controllable Text-to-Video Generationcited by 324
  11. 2023Ablating Concepts in Text-to-Image Diffusion Modelscited by 312
  12. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
  13. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  14. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  15. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 300
  16. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  17. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 264
  18. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 260
  19. 2022CelebV-HQ: A Large-Scale Video Facial Attributes Datasetcited by 246
  20. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  21. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 207
  22. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  23. 2023A Survey on Video Diffusion Modelscited by 203
  24. 2024PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generationcited by 185
  25. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 177
  26. 2023NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generationcited by 170
  27. 2023FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editingcited by 165
  28. 2026Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generationcited by 159
  29. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  30. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 153
  31. 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 151
  32. 2023VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planningcited by 138
  33. 2023Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generationcited by 137
  34. 2023Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Modelscited by 135
  35. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoisingcited by 135
  36. 2022Diffusion Models for Video Prediction and Infillingcited by 134
  37. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  38. 2022MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generationcited by 125

50 most cited of 253 All 253 (JSON)