# DreamArtist++: Controllable One-Shot Text-to-Image Generation via Positive-Negative Adapter

arXiv: https://arxiv.org/abs/2211.11337 · HTML: https://mlsanity.com/paper/2211.11337/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 38 of its 50 references that are in the map.
- First posted on arXiv 21 Nov 2022.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 38)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [Zero-Shot Text-Guided Object Generation with Dream Fields](https://mlsanity.com/paper/2112.01455/) (2021)

## Built on by (most cited first, 24 of 24)

- [ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs](https://mlsanity.com/paper/2311.13600/) (2023)
- [Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation](https://mlsanity.com/paper/2309.14859/) (2023)
- [TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models](https://mlsanity.com/paper/2403.17005/) (2024)
- [A Survey on Personalized Content Synthesis with Diffusion Models](https://mlsanity.com/paper/2405.05538/) (2024)
- [Edify Image: High-Quality Image Generation with Pixel Space Laplacian Diffusion Models](https://mlsanity.com/paper/2411.07126/) (2024)
- [CTRL-D: Controllable Dynamic 3D Scene Editing with Personalized 2D Diffusion](https://mlsanity.com/paper/2412.01792/) (2024)
- [DemoCaricature: Democratising Caricature Generation with a Rough Sketch](https://mlsanity.com/paper/2312.04364/) (2023)
- [DynamicID: Zero-Shot Multi-ID Image Personalization with Flexible Facial Editability](https://mlsanity.com/paper/2503.06505/) (2025)
- [CE-SDWV: Effective and Efficient Concept Erasure for Text-to-Image Diffusion Models via a Semantic-Driven Word Vocabulary](https://mlsanity.com/paper/2501.15562/) (2025)
- [Break-for-Make: Modular Low-Rank Adaptations for Composable Content-Style Customization](https://mlsanity.com/paper/2403.19456/) (2024)
- [Concept Conductor: Orchestrating Multiple Personalized Concepts in Text-to-Image Synthesis](https://mlsanity.com/paper/2408.03632/) (2024)
- [StyleForge: Enhancing Text-to-Image Synthesis for Any Artistic Styles with Dual Binding](https://mlsanity.com/paper/2404.05256/) (2024)
- [Anonymization Prompt Learning for Facial Privacy-Preserving Text-to-Image Generation](https://mlsanity.com/paper/2405.16895/) (2024)
- [BRAT: Bonus oRthogonAl Token for Architecture Agnostic Textual Inversion](https://mlsanity.com/paper/2408.04785/) (2024)
- [RoomEditor++: A Parameter-Sharing Diffusion Architecture for High-Fidelity Furniture Synthesis](https://mlsanity.com/paper/2512.17573/) (2025)
- [MULTI: Disentangling Camera Lens, Sensor, View, and Domain for Novel Image Generation](https://mlsanity.com/paper/2605.12134/) (2026)
- [LogicalDefender: Discovering, Extracting, and Utilizing Common-Sense Knowledge](https://mlsanity.com/paper/2403.11570/) (2024)
- [LEARN: A Story-Driven Layout-to-Image Generation Framework for STEM Instruction](https://mlsanity.com/paper/2508.11153/) (2025)
- [Finetuning-Free Personalization of Text to Image Generation via Hypernetworks](https://mlsanity.com/paper/2511.03156/) (2025)
- [Sissi: Zero-shot Style-guided Image Synthesis via Semantic-style Integration](https://mlsanity.com/paper/2601.06605/) (2026)
- [Rethinking Multi-Condition DiTs: Eliminating Redundant Attention via Position-Alignment and Keyword-Scoping](https://mlsanity.com/paper/2602.06850/) (2026)
- [Let Triggers Control: Frequency-Aware Dropout for Effective Token Control](https://mlsanity.com/paper/2603.27199/) (2026)
- [PostureObjectstitch: Anomaly Image Generation Considering Assembly Relationships in Industrial Scenarios](https://mlsanity.com/paper/2604.13863/) (2026)
- [Inline Critic Steers Image Editing](https://mlsanity.com/paper/2605.12724/) (2026)

## Abstract (arXiv)

State-of-the-arts text-to-image generation models such as Imagen and Stable Diffusion Model have succeed remarkable progresses in synthesizing high-quality, feature-rich images with high resolution guided by human text prompts. Since certain characteristics of image content e.g., very specific object entities or styles, are very hard to be accurately described by text, some example-based image generation approaches have been proposed, i.e. generating new concepts based on absorbing the salient features of a few input references. Despite of acknowledged successes, these methods have struggled on accurately capturing the reference examples' characteristics while keeping diverse and high-quality image generation, particularly in the one-shot scenario (i.e. given only one reference). To tackle this problem, we propose a simple yet effective framework, namely DreamArtist, which adopts a novel positive-negative prompt-tuning learning strategy on the pre-trained diffusion model, and it has shown to well handle the trade-off between the accurate controllability and fidelity of image generation with only one reference example. Specifically, our proposed framework incorporates both positive and negative embeddings or adapters and optimizes them in a joint manner. The positive part aggressively captures the salient characteristics of the reference image to drive diversified generation and the negative part rectifies inadequacies from the positive part. We have conducted extensive experiments and evaluated the proposed method from image similarity (fidelity) and diversity, generation controllability, and style cloning. And our DreamArtist has achieved a superior generation performance over existing methods. Besides, our additional evaluation on extended tasks, including concept compositions and prompt-guided image editing, demonstrates its effectiveness for more applications.