mlsanityData to 14 Oct 2026

M6-UFC: Unifying Multi-Modal Controls for Conditional Image Synthesis via Non-Autoregressive Generative Transformers

arXiv 2105.14211 ↗First posted 29 May 2021cs.CVcited by 36

Not on the map. Why?

Abstract (arXiv)

Conditional image synthesis aims to create an image according to some multi-modal guidance in the forms of textual descriptions, reference images, and image blocks to preserve, as well as their combinations. In this paper, instead of investigating these control signals separately, we propose a new two-stage architecture, M6-UFC, to unify any number of multi-modal controls. In M6-UFC, both the diverse control signals and the synthesized image are uniformly represented as a sequence of discrete tokens to be processed by Transformer. Different from existing two-stage autoregressive approaches such as DALL-E and VQGAN, M6-UFC adopts non-autoregressive generation (NAR) at the second stage to enhance the holistic consistency of the synthesized image, to support preserving specified image blocks, and to improve the synthesis speed. Further, we design a progressive algorithm that iteratively improves the non-autoregressively generated image, with the help of two estimators developed for evaluating the compliance with the controls and evaluating the fidelity of the synthesized image, respectively. Extensive experiments on a newly collected large-scale clothing dataset M2C-Fashion and a facial dataset Multi-Modal CelebA-HQ verify that M6-UFC can synthesize high-fidelity images that comply with flexible multi-modal controls.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 58 of its 71 references that are in the map. First posted on arXiv 29 May 2021.

Builds on

58 of 71 references.

References · 58

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  7. 2017Neural Discrete Representation Learningcited by 4,712
  8. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  11. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  12. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
38 more
  1. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  2. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  3. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  4. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  6. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  7. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  8. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,179
  9. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  10. 2016Conditional Image Generation with PixelCNN Decoderscited by 862
  11. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  12. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  13. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  14. 2018Free-Form Image Inpainting with Gated Convolutioncited by 594
  15. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 577
  16. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  17. 2017Non-Autoregressive Neural Machine Translationcited by 463
  18. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  19. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  20. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  21. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  22. 2018cGANs with Projection Discriminatorcited by 254
  23. 2019SEAN: Image Synthesis with Semantic Region-Adaptive Normalizationcited by 253
  24. 2017Pose Guided Person Image Generationcited by 251
  25. 2017StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 229
  26. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 228
  27. 2017Photographic Image Synthesis with Cascaded Refinement Networkscited by 223
  28. 2017GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibriumcited by 213
  29. 2018Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinementcited by 195
  30. 2019Controllable Text-to-Image Generationcited by 187
  31. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 185
  32. 2019ManiGAN: Text-Guided Image Manipulationcited by 156
  33. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 137
  34. 2021M6: A Chinese Multimodal Pretrainercited by 91
  35. 2018Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Languagecited by 90
  36. 2017Semantic Image Synthesis via Adversarial Learningcited by 87
  37. 2020X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformerscited by 59
  38. 2019Interactive Sketch & Fill: Multiclass Sketch-to-Image Translationcited by 51

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year

Citing papers per year: 2021: 4; 2022: 14; 2023: 11; 2024: 6; 2025: 1.

36 papers.

Sort:

By area:

  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  2. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  3. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  4. 2022MAGVIT: Masked Generative Video Transformercited by 346
  5. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 307
  6. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizercited by 262
  7. 2022Pretraining is All You Need for Image-to-Image Translationcited by 160
  8. 2023MMM: Generative Masked Motion Modelcited by 131
  9. 2023DiffCollage: Parallel Generation of Large Content with Diffusion Modelscited by 103
  10. 2024Survey of Cultural Awareness in Language Models: Text and Beyondcited by 103
  11. 2021Multimodal Conditional Image Synthesis with Product-of-Experts GANscited by 75
  12. 2022Visual Prompt Tuning for Generative Transfer Learningcited by 75
24 more
  1. 2023IconShop: Text-Guided Vector Icon Synthesis with Autoregressive Transformerscited by 75
  2. 2022NUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual Synthesiscited by 71
  3. 2022Improved Masked Image Generation with Token-Criticcited by 64
  4. 2024BAMM: Bidirectional Autoregressive Motion Modelcited by 55
  5. 2021Multimodal Image Synthesis and Editing: The Generative AI Eracited by 52
  6. 2021M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretrainingcited by 31
  7. 2023Improving Compositional Text-to-image Generation with Large Vision-Language Modelscited by 24
  8. 2023Modulating Pretrained Diffusion Models for Multimodal Image Synthesiscited by 23
  9. 2021EdiBERT, a generative model for image editingcited by 13
  10. 2023Towards High-Fidelity Text-Guided 3D Face Generation and Manipulation Using only Imagescited by 12
  11. 2023StraIT: Non-autoregressive Generation with Stratified Image Transformercited by 10
  12. 20223D-TOGO: Towards Text-Guided Cross-Category 3D Object Generationcited by 8
  13. 2024Masked Generative Priors Improve World Models Sequence Modelling Capabilitiescited by 7
  14. 2022Image Generation with Multimodal Priors using Denoising Diffusion Probabilistic Modelscited by 5
  15. 2022Unite and Conquer: Plug & Play Multi-Modal Synthesis using Diffusion Modelscited by 4
  16. 2023RealignDiff: Boosting Text-to-Image Diffusion Model with Coarse-to-fine Semantic Re-alignmentcited by 4
  17. 2022M6-Fashion: High-Fidelity Multi-modal Image Generation and Editingcited by 3
  18. 2024From Concrete to Abstract: A Multimodal Generative Approach to Abstract Concept Learningcited by 2
  19. 2022Spatially Multi-conditional Image Generationcited by 1
  20. 2022A Creative Industry Image Generation Dataset Based on Captionscited by 1
  21. 2023Emage: Non-Autoregressive Text-to-Image Generationcited by 1
  22. 2024Towards Multi-Task Multi-Modal Models: A Video Generative Perspectivecited by 0
  23. 2024AI Tailoring: Evaluating Influence of Image Features on Fashion Product Popularitycited by 0
  24. 2025Disentangling 3D from Large Vision-Language Models for Controlled Portrait Generationcited by 0