# PAI-Diffusion: Constructing and Serving a Family of Open Chinese Diffusion Models for Text-to-image Synthesis on the Cloud

arXiv: https://arxiv.org/abs/2309.05534 · HTML: https://mlsanity.com/paper/2309.05534/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 12 of its 18 references that are in the map.
- First posted on arXiv 11 Sep 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 12 of 12)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities](https://mlsanity.com/paper/2211.06679/) (2022)
- [DiffFashion: Reference-based Fashion Design with Structure-aware Transfer by Diffusion Models](https://mlsanity.com/paper/2302.06826/) (2023)
- [Optimal Linear Subspace Search: Learning to Construct Fast and High-Quality Schedulers for Diffusion Models](https://mlsanity.com/paper/2305.14677/) (2023)
- [DISC: A Dynamic Shape Compiler for Machine Learning Workloads](https://mlsanity.com/paper/2103.05288/) (2021)
- [EasyNLP: A Comprehensive and Easy-to-use Toolkit for Natural Language Processing](https://mlsanity.com/paper/2205.00258/) (2022)
- [Efficient Cross-Lingual Transfer for Chinese Stable Diffusion with Images as Pivots](https://mlsanity.com/paper/2305.11540/) (2023)
- [Design Booster: A Text-Guided Diffusion Model for Image Translation with Spatial Layout Preservation](https://mlsanity.com/paper/2302.02284/) (2023)

## Built on by (most cited first, 5 of 5)

- [Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding](https://mlsanity.com/paper/2405.08748/) (2024)
- [Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing](https://mlsanity.com/paper/2403.03431/) (2024)
- [Efficient Diffusion Models: A Comprehensive Survey from Principles to Practices](https://mlsanity.com/paper/2410.11795/) (2024)
- [Taiyi-Diffusion-XL: Advancing Bilingual Text-to-Image Generation with Large Vision-Language Model Support](https://mlsanity.com/paper/2401.14688/) (2024)
- [JuZhou 1.0 Technical Report: The First Edge-Native Text-to-Image Foundation Model Trained Entirely on China-Developed AI Accelerators](https://mlsanity.com/paper/2606.28421/) (2026)

## Abstract (arXiv)

Text-to-image synthesis for the Chinese language poses unique challenges due to its large vocabulary size, and intricate character relationships. While existing diffusion models have shown promise in generating images from textual descriptions, they often neglect domain-specific contexts and lack robustness in handling the Chinese language. This paper introduces PAI-Diffusion, a comprehensive framework that addresses these limitations. PAI-Diffusion incorporates both general and domain-specific Chinese diffusion models, enabling the generation of contextually relevant images. It explores the potential of using LoRA and ControlNet for fine-grained image style transfer and image editing, empowering users with enhanced control over image generation. Moreover, PAI-Diffusion seamlessly integrates with Alibaba Cloud's Machine Learning Platform for AI, providing accessible and scalable solutions. All the Chinese diffusion model checkpoints, LoRAs, and ControlNets, including domain-specific ones, are publicly available. A user-friendly Chinese WebUI and the diffusers-api elastic inference toolkit, also open-sourced, further facilitate the easy deployment of PAI-Diffusion models in various environments, making it a valuable resource for Chinese text-to-image synthesis.