# OneActor: Consistent Character Generation via Cluster-Conditioned Guidance arXiv: https://arxiv.org/abs/2404.10267 ยท HTML: https://mlsanity.com/paper/2404.10267/ ## Key facts - Cited by 15 papers in the map (data through 14 Oct 2026). - Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. - Builds on 39 of its 45 references that are in the map. - First posted on arXiv 16 Apr 2024. ## Placed in - Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/) ## Builds on (most cited first, 30 of 39) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018) - [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020) - [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023) - [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023) - [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015) - [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022) - [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023) - [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023) - [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022) - [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021) - [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022) - [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017) - [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023) - [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022) - [Zero-shot Image-to-Image Translation](https://mlsanity.com/paper/2302.03027/) (2023) - [BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing](https://mlsanity.com/paper/2305.14720/) (2023) - [ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation](https://mlsanity.com/paper/2302.13848/) (2023) - [SVDiff: Compact Parameter Space for Diffusion Fine-Tuning](https://mlsanity.com/paper/2303.11305/) (2023) - [DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data](https://mlsanity.com/paper/2306.09344/) (2023) - [PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding](https://mlsanity.com/paper/2312.04461/) (2023) - [Break-A-Scene: Extracting Multiple Concepts from a Single Image](https://mlsanity.com/paper/2305.16311/) (2023) - [Subject-driven Text-to-Image Generation via Apprenticeship Learning](https://mlsanity.com/paper/2304.00186/) (2023) - [Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models](https://mlsanity.com/paper/2302.12228/) (2023) - [Key-Locked Rank One Editing for Text-to-Image Personalization](https://mlsanity.com/paper/2305.01644/) (2023) - [Training-Free Consistent Text-to-Image Generation](https://mlsanity.com/paper/2402.03286/) (2024) ## Built on by (most cited first, 15 of 15) - [Large-Scale Text-to-Image Model with Inpainting is a Zero-Shot Subject-Driven Image Generator](https://mlsanity.com/paper/2411.15466/) (2024) - [Schedule Your Edit: A Simple yet Effective Diffusion Noise Schedule for Image Editing](https://mlsanity.com/paper/2410.18756/) (2024) - [StoryMaker: Towards Holistic Consistent Characters in Text-to-image Generation](https://mlsanity.com/paper/2409.12576/) (2024) - [EchoShot: Multi-Shot Portrait Video Generation](https://mlsanity.com/paper/2506.15838/) (2025) - [SpotActor: Training-Free Layout-Controlled Consistent Image Generation](https://mlsanity.com/paper/2409.04801/) (2024) - [DynamicID: Zero-Shot Multi-ID Image Personalization with Flexible Facial Editability](https://mlsanity.com/paper/2503.06505/) (2025) - [TAG-MoE: Task-Aware Gating for Unified Generative Mixture-of-Experts](https://mlsanity.com/paper/2601.08881/) (2026) - [PlanViz: Evaluating Planning-Oriented Image Generation and Editing for Computer-Use Tasks](https://mlsanity.com/paper/2602.06663/) (2026) - [Improving Multi-Subject Consistency in Open-Domain Image Generation with Isolation and Reposition Attention](https://mlsanity.com/paper/2411.19261/) (2024) - [MotionWeaver: Holistic 4D-Anchored Framework for Multi-Humanoid Image Animation](https://mlsanity.com/paper/2602.13326/) (2026) - [Infinite-Story: A Training-Free Consistent Text-to-Image Generation](https://mlsanity.com/paper/2511.13002/) (2025) - [CoCoIns: Consistent Subject Generation via Contrastive Instantiated Concepts](https://mlsanity.com/paper/2503.24387/) (2025) - [FontCrafter: High-Fidelity Element-Driven Artistic Font Creation with Visual In-Context Generation](https://mlsanity.com/paper/2603.22054/) (2026) - [DepthArb: Training-Free Depth-Arbitrated Generation for Occlusion-Robust Image Synthesis](https://mlsanity.com/paper/2603.23924/) (2026) - [AnyID: Ultra-Fidelity Universal Identity-Preserving Video Generation from Any Visual References](https://mlsanity.com/paper/2603.25188/) (2026) ## Abstract (arXiv) Text-to-image diffusion models benefit artists with high-quality image generation. Yet their stochastic nature hinders artists from creating consistent images of the same subject. Existing methods try to tackle this challenge and generate consistent content in various ways. However, they either depend on external restricted data or require expensive tuning of the diffusion model. For this issue, we propose a novel one-shot tuning paradigm, termed OneActor. It efficiently performs consistent subject generation solely driven by prompts via a learned semantic guidance to bypass the laborious backbone tuning. We lead the way to formalize the objective of consistent subject generation from a clustering perspective, and thus design a cluster-conditioned model. To mitigate the overfitting challenge shared by one-shot tuning pipelines, we augment the tuning with auxiliary samples and devise two inference strategies: semantic interpolation and cluster guidance. These techniques are later verified to significantly improve the generation quality. Comprehensive experiments show that our method outperforms a variety of baselines with satisfactory subject consistency, superior prompt conformity as well as high image quality. Our method is capable of multi-subject generation and compatible with popular diffusion extensions. Besides, we achieve a 4 times faster tuning speed than tuning-based baselines and, if desired, avoid increasing the inference time. Furthermore, our method can be naturally utilized to pre-train a consistent subject generation network from scratch, which will implement this research task into more practical applications. (Project page: https://johnneywang.github.io/OneActor-webpage/)