# InstancePin: Instance-Addressable Layout-to-Image Diffusion via Coordinate Pinning

arXiv: https://arxiv.org/abs/2608.00588 · HTML: https://mlsanity.com/paper/2608.00588/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 30 of its 35 references that are in the map.
- First posted on arXiv 1 Aug 2026.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2305.16322/) (2023)
- [Dilated Residual Networks](https://mlsanity.com/paper/1705.09914/) (2017)
- [Learning to Discover Cross-Domain Relations with Generative Adversarial Networks](https://mlsanity.com/paper/1703.05192/) (2017)
- [SpaText: Spatio-Textual Representation for Controllable Image Generation](https://mlsanity.com/paper/2211.14305/) (2022)
- [Pretraining is All You Need for Image-to-Image Translation](https://mlsanity.com/paper/2205.12952/) (2022)
- [Semantic Image Synthesis via Diffusion Models](https://mlsanity.com/paper/2207.00050/) (2022)
- [You Only Need Adversarial Supervision for Semantic Image Synthesis](https://mlsanity.com/paper/2012.04781/) (2020)
- [Freestyle Layout-to-Image Synthesis](https://mlsanity.com/paper/2303.14412/) (2023)
- [Semantically Multi-modal Image Synthesis](https://mlsanity.com/paper/2003.12697/) (2020)
- [Efficient Semantic Image Synthesis via Class-Adaptive Normalization](https://mlsanity.com/paper/2012.04644/) (2020)
- [Diverse Semantic Image Synthesis via Probability Distribution Modeling](https://mlsanity.com/paper/2103.06878/) (2021)
- [Image Synthesis via Semantic Composition](https://mlsanity.com/paper/2109.07053/) (2021)
- [PLACE: Adaptive Layout-Semantic Fusion for Semantic Image Synthesis](https://mlsanity.com/paper/2403.01852/) (2024)
- [Decoupled Textual Embeddings for Customized Image Generation](https://mlsanity.com/paper/2312.11826/) (2023)
- [Unlocking Pre-trained Image Backbones for Semantic Image Synthesis](https://mlsanity.com/paper/2312.13314/) (2023)

## Abstract (arXiv)

Layout-to-image diffusion models have achieved impressive semantic controllability by conditioning generation on category-level segmentation maps. However, such category-aligned control is not necessarily instance-addressable: multiple nearby objects from the same category are often treated as a shared semantic region, leading to ambiguous boundaries, averaged appearances, and feature confusion among instances. This limitation is particularly evident in urban scene synthesis, where small and crowded pedestrians or vehicles require fine-grained instance separation while preserving global scene consistency. In this paper, we propose InstancePin, an instance-addressable layout-to-image diffusion framework that pins each object instance with an explicit coordinate anchor. Instead of directly injecting instance masks into the pretrained backbone, InstancePin introduces an independent instance-aware adapter to preserve the category-level generation prior while learning instance-specific spatial control. For each instance, its center coordinate is encoded with Fourier features and projected into a coordinate token, which serves as a spatial anchor queried by latent image features through coordinate pinning attention. To make these anchors spatially meaningful, we further supervise the coordinate attention maps with instance regions, encouraging each coordinate token to activate its corresponding object area. Finally, an instance-mask guided fusion module routes pretrained backbone features to non-instance regions and adapter features to instance regions, enabling local instance refinement without sacrificing global semantic fidelity. Extensive experiments on Cityscapes demonstrate that InstancePin mitigates instance entanglement in dense layouts and improves both image fidelity and semantic consistency.