# CP2: Copy-Paste Contrastive Pretraining for Semantic Segmentation

arXiv: https://arxiv.org/abs/2203.11709 · HTML: https://mlsanity.com/paper/2203.11709/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 55 references that are in the map.
- First posted on arXiv 22 Mar 2022.

## Builds on (most cited first, 30 of 46)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)

## Built on by (most cited first, 26 of 26)

- [SCLIP: Rethinking Self-Attention for Dense Vision-Language Inference](https://mlsanity.com/paper/2312.01597/) (2023)
- [VICRegL: Self-Supervised Learning of Local Visual Features](https://mlsanity.com/paper/2210.01571/) (2022)
- [Bidirectional Copy-Paste for Semi-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2305.00673/) (2023)
- [MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Features](https://mlsanity.com/paper/2307.12698/) (2023)
- [A Survey on Label-efficient Deep Image Segmentation: Bridging the Gap between Weak Supervision and Dense Prediction](https://mlsanity.com/paper/2207.01223/) (2022)
- [Learning to Decompose Visual Features with Latent Textual Prompts](https://mlsanity.com/paper/2210.04287/) (2022)
- [CrOC: Cross-View Online Clustering for Dense Visual Representation Learning](https://mlsanity.com/paper/2303.13245/) (2023)
- [CrIBo: Self-Supervised Learning via Cross-Image Object-Level Bootstrapping](https://mlsanity.com/paper/2310.07855/) (2023)
- [How far can we go with ImageNet for Text-to-Image generation?](https://mlsanity.com/paper/2502.21318/) (2025)
- [Introspective Deep Metric Learning](https://mlsanity.com/paper/2309.09982/) (2023)
- [INoD: Injected Noise Discriminator for Self-Supervised Representation Learning in Agricultural Fields](https://mlsanity.com/paper/2303.18101/) (2023)
- [Revisiting DETR Pre-training for Object Detection](https://mlsanity.com/paper/2308.01300/) (2023)
- [OPERA: Omni-Supervised Representation Learning with Hierarchical Supervisions](https://mlsanity.com/paper/2210.05557/) (2022)
- [A Systematic Literature Review of Computer Vision Applications in Robotized Wire Harness Assembly](https://mlsanity.com/paper/2309.13744/) (2023)
- [Disentangled Pre-training for Image Matting](https://mlsanity.com/paper/2304.00784/) (2023)
- [Self-supervised Learning of Contextualized Local Visual Embeddings](https://mlsanity.com/paper/2310.00527/) (2023)
- [SemanticMIM: Marring Masked Image Modeling with Semantics Compression for General Visual Representation](https://mlsanity.com/paper/2406.10673/) (2024)
- [Investigating Self-Supervised Methods for Label-Efficient Learning](https://mlsanity.com/paper/2406.17460/) (2024)
- [Self-Supervised Learning for Real-World Object Detection: a Survey](https://mlsanity.com/paper/2410.07442/) (2024)
- [Copy-Pasting Coherent Depth Regions Improves Contrastive Learning for Urban-Scene Segmentation](https://mlsanity.com/paper/2211.14074/) (2022)
- [MIMIC: Masked Image Modeling with Image Correspondences](https://mlsanity.com/paper/2306.15128/) (2023)
- [Dual-Teacher Ensemble Models with Double-Copy-Paste for 3D Semi-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2410.11509/) (2024)
- [Beyond the Encoder: Joint Encoder-Decoder Contrastive Pre-Training Improves Dense Prediction](https://mlsanity.com/paper/2503.17526/) (2025)
- [Tracing Copied Pixels and Regularizing Patch Affinity in Copy Detection](https://mlsanity.com/paper/2602.17484/) (2026)
- [Mask-supervised Object-centric Representation Learning with LeJEPA](https://mlsanity.com/paper/2607.02404/) (2026)
- [Depth-Guided Contrastive Learning for 2D Representations with 3D Spatial Awareness](https://mlsanity.com/paper/2609.28159/) (2026)

## Abstract (arXiv)

Recent advances in self-supervised contrastive learning yield good image-level representation, which favors classification tasks but usually neglects pixel-level detailed information, leading to unsatisfactory transfer performance to dense prediction tasks such as semantic segmentation. In this work, we propose a pixel-wise contrastive learning method called CP2 (Copy-Paste Contrastive Pretraining), which facilitates both image- and pixel-level representation learning and therefore is more suitable for downstream dense prediction tasks. In detail, we copy-paste a random crop from an image (the foreground) onto different background images and pretrain a semantic segmentation model with the objective of 1) distinguishing the foreground pixels from the background pixels, and 2) identifying the composed images that share the same foreground.Experiments show the strong performance of CP2 in downstream semantic segmentation: By finetuning CP2 pretrained models on PASCAL VOC 2012, we obtain 78.6% mIoU with a ResNet-50 and 79.5% with a ViT-S.