# Object-level Visual Prompts for Compositional Image Generation arXiv: https://arxiv.org/abs/2501.01424 ยท HTML: https://mlsanity.com/paper/2501.01424/ ## Key facts - Cited by 14 papers in the map (data through 14 Oct 2026). - Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. - Builds on 42 of its 57 references that are in the map. - First posted on arXiv 2 Jan 2025. ## Placed in - Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/) ## Builds on (most cited first, 30 of 42) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018) - [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020) - [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021) - [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023) - [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022) - [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022) - [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016) - [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014) - [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023) - [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023) - [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022) - [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022) - [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021) - [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021) - [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023) - [Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2301.13826/) (2023) - [Zero-shot Image-to-Image Translation](https://mlsanity.com/paper/2302.03027/) (2023) - [BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing](https://mlsanity.com/paper/2305.14720/) (2023) - [AnyDoor: Zero-shot Object-level Image Customization](https://mlsanity.com/paper/2307.09481/) (2023) - [InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning](https://mlsanity.com/paper/2304.03411/) (2023) - [Training-Free Layout Control with Cross-Attention Guidance](https://mlsanity.com/paper/2304.03373/) (2023) - [FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention](https://mlsanity.com/paper/2305.10431/) (2023) - [Toward Multimodal Image-to-Image Translation](https://mlsanity.com/paper/1711.11586/) (2017) - [Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models](https://mlsanity.com/paper/2305.18292/) (2023) - [Break-A-Scene: Extracting Multiple Concepts from a Single Image](https://mlsanity.com/paper/2305.16311/) (2023) - [Large Scale Image Completion via Co-Modulated Generative Adversarial Networks](https://mlsanity.com/paper/2103.10428/) (2021) - [Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models](https://mlsanity.com/paper/2302.12228/) (2023) - [Key-Locked Rank One Editing for Text-to-Image Personalization](https://mlsanity.com/paper/2305.01644/) (2023) ## Built on by (most cited first, 14 of 14) - [Generating Multi-Image Synthetic Data for Text-to-Image Customization](https://mlsanity.com/paper/2502.01720/) (2025) - [Personalize Anything for Free with Diffusion Transformer](https://mlsanity.com/paper/2503.12590/) (2025) - [Personalized Image Generation with Deep Generative Models: A Decade Survey](https://mlsanity.com/paper/2502.13081/) (2025) - [WithAnyone: Towards Controllable and ID Consistent Image Generation](https://mlsanity.com/paper/2510.14975/) (2025) - [Tight Inversion: Image-Conditioned Inversion for Real Image Editing](https://mlsanity.com/paper/2502.20376/) (2025) - [ComposeMe: Attribute-Specific Image Prompts for Controllable Human Image Generation](https://mlsanity.com/paper/2509.18092/) (2025) - [Piece it Together: Part-Based Concepting with IP-Priors](https://mlsanity.com/paper/2503.10365/) (2025) - [Tuning-free Visual Effect Transfer across Videos](https://mlsanity.com/paper/2601.07833/) (2026) - [CPAM: Context-Preserving Adaptive Manipulation for Zero-Shot Real Image Editing](https://mlsanity.com/paper/2506.18438/) (2025) - [Neural USD: An object-centric framework for iterative editing and control](https://mlsanity.com/paper/2510.23956/) (2025) - [Doc-CoB: Enhancing Document Understanding with Visual Chain-of-Boxes Reasoning](https://mlsanity.com/paper/2505.18603/) (2025) - [Griffin: Generative Reference and Layout Guided Image Composition](https://mlsanity.com/paper/2509.23643/) (2025) - [TokenAR: Multiple Subject Generation via Autoregressive Token-level enhancement](https://mlsanity.com/paper/2510.16332/) (2025) - [Muses: Designing, Composing, Generating Nonexistent Fantasy 3D Creatures without Training](https://mlsanity.com/paper/2601.03256/) (2026) ## Abstract (arXiv) We introduce a method for composing object-level visual prompts within a text-to-image diffusion model. Our approach addresses the task of generating semantically coherent compositions across diverse scenes and styles, similar to the versatility and expressiveness offered by text prompts. A key challenge in this task is to preserve the identity of the objects depicted in the input visual prompts, while also generating diverse compositions across different images. To address this challenge, we introduce a new KV-mixed cross-attention mechanism, in which keys and values are learned from distinct visual representations. The keys are derived from an encoder with a small bottleneck for layout control, whereas the values come from a larger bottleneck encoder that captures fine-grained appearance details. By mixing keys and values from these complementary sources, our model preserves the identity of the visual prompts while supporting flexible variations in object arrangement, pose, and composition. During inference, we further propose object-level compositional guidance to improve the method's identity preservation and layout correctness. Results show that our technique produces diverse scene compositions that preserve the unique characteristics of each visual prompt, expanding the creative potential of text-to-image generation.