# X-Paste: Revisiting Scalable Copy-Paste for Instance Segmentation using CLIP and StableDiffusion

arXiv: https://arxiv.org/abs/2212.03863 · HTML: https://mlsanity.com/paper/2212.03863/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 64 references that are in the map.
- First posted on arXiv 7 Dec 2022.

## Builds on (most cited first, 30 of 57)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://mlsanity.com/paper/2111.14822/) (2021)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://mlsanity.com/paper/2012.07177/) (2020)
- [CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows](https://mlsanity.com/paper/2107.00652/) (2021)
- [Image Segmentation Using Text and Image Prompts](https://mlsanity.com/paper/2112.10003/) (2021)
- [Open-Vocabulary Object Detection Using Captions](https://mlsanity.com/paper/2011.10678/) (2020)
- [U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection](https://mlsanity.com/paper/2005.09007/) (2020)
- [Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation](https://mlsanity.com/paper/2206.02777/) (2022)
- [Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model](https://mlsanity.com/paper/2203.14940/) (2022)
- [Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection](https://mlsanity.com/paper/1708.01642/) (2017)

## Built on by (most cited first, 30 of 49)

- [InstanceDiffusion: Instance-level Control for Image Generation](https://mlsanity.com/paper/2402.03290/) (2024)
- [Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding](https://mlsanity.com/paper/2312.00081/) (2023)
- [MosaicFusion: Diffusion Models as Data Augmenters for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/2309.13042/) (2023)
- [Image Segmentation in Foundation Model Era: A Survey](https://mlsanity.com/paper/2408.12957/) (2024)
- [AeroGen: Enhancing Remote Sensing Object Detection with Diffusion-Driven Data Generation](https://mlsanity.com/paper/2411.15497/) (2024)
- [SmartEraser: Remove Anything from Images using Masked-Region Guidance](https://mlsanity.com/paper/2501.08279/) (2025)
- [HairCLIPv2: Unifying Hair Editing via Proxy Feature Blending](https://mlsanity.com/paper/2310.10651/) (2023)
- [ODGEN: Domain-specific Object Detection Data Generation with Diffusion Models](https://mlsanity.com/paper/2405.15199/) (2024)
- [Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation](https://mlsanity.com/paper/2410.02369/) (2024)
- [DiverGen: Improving Instance Segmentation by Learning Wider Data Distribution with More Diverse Generative Data](https://mlsanity.com/paper/2405.10185/) (2024)
- [Domain-RAG: Retrieval-Guided Compositional Image Generation for Cross-Domain Few-Shot Object Detection](https://mlsanity.com/paper/2506.05872/) (2025)
- [PIN: Positional Insert Unlocks Object Localisation Abilities in VLMs](https://mlsanity.com/paper/2402.08657/) (2024)
- [Generative Enhancement for 3D Medical Images](https://mlsanity.com/paper/2403.12852/) (2024)
- [Gen2Det: Generate to Detect](https://mlsanity.com/paper/2312.04566/) (2023)
- [Generative Active Learning for Long-tailed Instance Segmentation](https://mlsanity.com/paper/2406.02435/) (2024)
- [How far can we go with ImageNet for Text-to-Image generation?](https://mlsanity.com/paper/2502.21318/) (2025)
- [SMILE: Infusing Spatial and Motion Semantics in Masked Video Learning](https://mlsanity.com/paper/2504.00527/) (2025)
- [Unlocking Pre-trained Image Backbones for Semantic Image Synthesis](https://mlsanity.com/paper/2312.13314/) (2023)
- [ReCon: Region-Controllable Data Augmentation with Rectification and Alignment for Object Detection](https://mlsanity.com/paper/2510.15783/) (2025)
- [Language-Guided Instance-Aware Domain-Adaptive Panoptic Segmentation](https://mlsanity.com/paper/2404.03799/) (2024)
- [Auto Cherry-Picker: Learning from High-quality Generative Data Driven by Language](https://mlsanity.com/paper/2406.20085/) (2024)
- [LocoMotion: Learning Motion-Focused Video-Language Representations](https://mlsanity.com/paper/2410.12018/) (2024)
- [Imagining the Unseen: Generative Location Modeling for Object Placement](https://mlsanity.com/paper/2410.13564/) (2024)
- [MonoPlace3D: Learning 3D-Aware Object Placement for 3D Monocular Detection](https://mlsanity.com/paper/2504.06801/) (2025)
- [Unbiased Object Detection Beyond Frequency with Visually Prompted Image Synthesis](https://mlsanity.com/paper/2510.18229/) (2025)
- [Frequency-based Matcher for Long-tailed Semantic Segmentation](https://mlsanity.com/paper/2406.03917/) (2024)
- [Gen-n-Val: Agentic Image Data Generation and Validation](https://mlsanity.com/paper/2506.04676/) (2025)
- [Enrich the content of the image Using Context-Aware Copy Paste](https://mlsanity.com/paper/2407.08151/) (2024)
- [Better (pseudo-)labels for semi-supervised instance segmentation](https://mlsanity.com/paper/2403.11675/) (2024)
- [Add-SD: Rational Generation without Manual Reference](https://mlsanity.com/paper/2407.21016/) (2024)

## Abstract (arXiv)

Copy-Paste is a simple and effective data augmentation strategy for instance segmentation. By randomly pasting object instances onto new background images, it creates new training data for free and significantly boosts the segmentation performance, especially for rare object categories. Although diverse, high-quality object instances used in Copy-Paste result in more performance gain, previous works utilize object instances either from human-annotated instance segmentation datasets or rendered from 3D object models, and both approaches are too expensive to scale up to obtain good diversity. In this paper, we revisit Copy-Paste at scale with the power of newly emerged zero-shot recognition models (e.g., CLIP) and text2image models (e.g., StableDiffusion). We demonstrate for the first time that using a text2image model to generate images or zero-shot recognition model to filter noisily crawled images for different object categories is a feasible way to make Copy-Paste truly scalable. To make such success happen, we design a data acquisition and processing framework, dubbed ``X-Paste", upon which a systematic study is conducted. On the LVIS dataset, X-Paste provides impressive improvements over the strong baseline CenterNet2 with Swin-L as the backbone. Specifically, it archives +2.6 box AP and +2.1 mask AP gains on all classes and even more significant gains with +6.8 box AP, +6.5 mask AP on long-tail classes. Our code and models are available at https://github.com/yoctta/XPaste.