mlsanityData to 7 Oct 2026

Pretraining is All You Need for Image-to-Image Translation

arXiv 2205.12952 ↗First posted 25 May 2022cs.CVcited by 160

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

We propose to use pretraining to boost general image-to-image translation. Prior image-to-image translation methods usually need dedicated architectural design and train individual translation models from scratch, struggling for high-quality generation of complex scenes, especially when paired training data are not abundant. In this paper, we regard each image-to-image translation problem as a downstream task and introduce a simple and generic framework that adapts a pretrained diffusion model to accommodate various kinds of image-to-image translation. We also propose adversarial training to enhance the texture synthesis in the diffusion model training, in conjunction with normalized guidance sampling to improve the generation quality. We present extensive empirical comparison across various tasks on challenging benchmarks such as ADE20K, COCO-Stuff, and DIODE, showing the proposed pretraining-based image-to-image translation (PITI) is capable of synthesizing images of unprecedented realism and faithfulness.

Key facts. Cited by 160 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 52 of its 62 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

52 of 62 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  3. 2021Palette: Image-to-Image Diffusion Models10 Nov

Other references · 49

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  12. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
37 more
  1. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  4. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  5. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  7. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  8. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  9. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  10. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  11. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,036
  12. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
  13. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  14. 2020Pre-Trained Image Processing Transformercited by 891
  15. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  16. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  17. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  18. 2015Holistically-Nested Edge Detectioncited by 590
  19. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  20. 2021Designing a Practical Degradation Model for Deep Blind Image Super-Resolutioncited by 524
  21. 2020PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Modelscited by 358
  22. 2021Towards Real-World Blind Face Restoration with Generative Facial Priorcited by 347
  23. 2017Toward Multimodal Image-to-Image Translationcited by 295
  24. 2019DIODE: A Dense Indoor and Outdoor DEpth Datasetcited by 236
  25. 2020Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulationcited by 225
  26. 2022The Role of ImageNet Classes in Fr'echet Inception Distancecited by 211
  27. 2021GAN Prior Embedded Network for Blind Face Restoration in the Wildcited by 197
  28. 2021High-Fidelity GAN Inversion for Image Attribute Editingcited by 197
  29. 2020Editing in Style: Uncovering the Local Semantics of GANscited by 183
  30. 2021StyleSwin: Transformer-based GAN for High-resolution Image Generationcited by 152
  31. 2020You Only Need Adversarial Supervision for Semantic Image Synthesiscited by 129
  32. 2020Cross-domain Correspondence Learning for Exemplar-based Image Translationcited by 116
  33. 2021Image-to-Image Translation: Methods and Applicationscited by 116
  34. 2021Multimodal Conditional Image Synthesis with Product-of-Experts GANscited by 75
  35. 2020Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applicationscited by 48
  36. 2021M6-UFC: Unifying Multi-Modal Controls for Conditional Image Synthesis via Non-Autoregressive Generative Transformerscited by 36
  37. 2021Multimodal Variational Autoencoders for Semi-Supervised Learning: In Defense of Product-of-Expertscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 17; 2023: 75; 2024: 45; 2025: 16; 2026: 7.

160 papers.

Sort:

By area:

  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  2. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  3. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  4. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  5. 2022Diffusion Models in Vision: A Surveycited by 1,008
  6. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  7. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  8. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 643
  9. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 529
  10. 2023Zero-shot Image-to-Image Translationcited by 503
  11. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Modelscited by 489
  12. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
38 more
  1. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Priorcited by 341
  2. 2023Pix2Video: Video Editing using Image Diffusioncited by 318
  3. 2023Ablating Concepts in Text-to-Image Diffusion Modelscited by 312
  4. 2022Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusioncited by 309
  5. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 303
  6. 2022SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Modelcited by 276
  7. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 264
  8. 2023I2SB: Image-to-Image Schr"odinger Bridgecited by 214
  9. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 207
  10. 2023UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wildcited by 197
  11. 2023FreeU: Free Lunch in Diffusion U-Netcited by 195
  12. 2022Sketch-Guided Text-to-Image Diffusion Modelscited by 180
  13. 2024U-KAN Makes Strong Backbone for Medical Image Segmentation and Generationcited by 176
  14. 2023Dense Text-to-Image Generation with Attention Modulationcited by 162
  15. 2022Improving Sample Quality of Diffusion Models Using Self-Attention Guidancecited by 151
  16. 2023LaDI-VTON: Latent Diffusion Textual-Inversion Enhanced Virtual Try-Oncited by 151
  17. 2022Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decodingcited by 142
  18. 2023Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generationcited by 133
  19. 2023DCFace: Synthetic Face Generation with Dual Condition Diffusion Modelcited by 116
  20. 2023Collaborative Diffusion for Multi-Modal Face Generation and Editingcited by 116
  21. 2024One-Step Image Translation with Text-to-Image Modelscited by 111
  22. 2023CoDeF: Content Deformation Fields for Temporally Consistent Video Processingcited by 108
  23. 2024MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizercited by 80
  24. 2023Ranni: Taming Text-to-Image Diffusion for Accurate Instruction Followingcited by 77
  25. 2024Anatomically-Controllable Medical Image Generation with Segmentation-Guided Diffusion Modelscited by 69
  26. 2023LooseControl: Lifting ControlNet for Generalized Depth Conditioningcited by 68
  27. 2023Freestyle Layout-to-Image Synthesiscited by 67
  28. 2023Zero-shot spatial layout conditioning for text-to-image diffusion modelscited by 64
  29. 2023Consistent123: One Image to Highly Consistent 3D Asset Using Case-Aware Diffusion Priorscited by 64
  30. 2023Guided Image Synthesis via Initial Image Editing in Diffusion Modelcited by 63
  31. 2023Multimodal Garment Designer: Human-Centric Latent Diffusion Models for Fashion Image Editingcited by 62
  32. 2023ZONE: Zero-Shot Instruction-Guided Local Editingcited by 61
  33. 2022Diffusion-Based Scene Graph to Image Generation with Masked Contrastive Pre-Trainingcited by 56
  34. 2023Relightful Harmonization: Lighting-aware Portrait Background Replacementcited by 56
  35. 2024FlashTex: Fast Relightable Mesh Texturing with LightControlNetcited by 56
  36. 2023Diff2Lip: Audio Conditioned Diffusion Models for Lip-Synchronizationcited by 49
  37. 2023WaveDM: Wavelet-Based Diffusion Models for Image Restorationcited by 46
  38. 2023Create Your World: Lifelong Text-to-Image Diffusioncited by 46

50 most cited of 160 All 160 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Image-to-Image Translation with Diffusion Transformers and CLIP-Based Image Conditioning
  2. 2024Diffusion-Based Image-to-Image Translation by Noise Correction via Prompt Interpolation
  3. 2023Improving Diffusion-based Image Translation using Asymmetric Gradient Guidance
  4. 2025A Diffusion Model Translator for Efficient Image-to-Image Translation
  5. 2024Seed-to-Seed: Unpaired Image Translation in Diffusion Seed Space
  6. 2023DiffI2I: Efficient Diffusion Model for Image-to-Image Translation
  7. 2025d-Sketch: Improving Visual Fidelity of Sketch-to-Image Translation with Pretrained Latent Diffusion Models without Retraining
  8. 2023Design Booster: A Text-Guided Diffusion Model for Image Translation with Spatial Layout Preservation