# CASTing Your Model: Learning to Localize Improves Self-Supervised Representations

arXiv: https://arxiv.org/abs/2012.04630 · HTML: https://mlsanity.com/paper/2012.04630/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Dec 2020.

## Built on by (most cited first, 30 of 63)

- [Efficient Visual Pretraining with Contrastive Detection](https://mlsanity.com/paper/2103.10957/) (2021)
- [Unsupervised Object-Level Representation Learning from Scene Images](https://mlsanity.com/paper/2106.11952/) (2021)
- [Self-Supervised Visual Representation Learning with Semantic Grouping](https://mlsanity.com/paper/2205.15288/) (2022)
- [Crafting Better Contrastive Views for Siamese Representation Learning](https://mlsanity.com/paper/2202.03278/) (2022)
- [Hyperbolic Contrastive Learning for Visual Representations beyond Objects](https://mlsanity.com/paper/2212.00653/) (2022)
- [Revisiting Contrastive Methods for Unsupervised Learning of Visual Representations](https://mlsanity.com/paper/2106.05967/) (2021)
- [Unsupervised Hierarchical Semantic Segmentation with Multiview Cosegmentation and Clustering Transformers](https://mlsanity.com/paper/2204.11432/) (2022)
- [Object-aware Contrastive Learning for Debiased Scene Representation](https://mlsanity.com/paper/2108.00049/) (2021)
- [CLIP-S4: Language-Guided Self-Supervised Semantic Segmentation](https://mlsanity.com/paper/2305.01040/) (2023)
- [Learning Visual Representations via Language-Guided Sampling](https://mlsanity.com/paper/2302.12248/) (2023)
- [UniVIP: A Unified Framework for Self-Supervised Visual Pre-training](https://mlsanity.com/paper/2203.06965/) (2022)
- [Architecture-Agnostic Masked Image Modeling -- From ViT back to CNN](https://mlsanity.com/paper/2205.13943/) (2022)
- [EAGLE: Eigen Aggregation Learning for Object-Centric Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2403.01482/) (2024)
- [Anatomical Invariance Modeling and Semantic Alignment for Self-supervised Learning in 3D Medical Image Analysis](https://mlsanity.com/paper/2302.05615/) (2023)
- [Augmentations in Graph Contrastive Learning: Current Methodological Flaws & Towards Better Practices](https://mlsanity.com/paper/2111.03220/) (2021)
- [Improving Visual Grounding by Encouraging Consistent Gradient-based Explanations](https://mlsanity.com/paper/2206.15462/) (2022)
- [Object-Aware Cropping for Self-Supervised Learning](https://mlsanity.com/paper/2112.00319/) (2021)
- [CrOC: Cross-View Online Clustering for Dense Visual Representation Learning](https://mlsanity.com/paper/2303.13245/) (2023)
- [Boosting Video Object Segmentation via Space-time Correspondence Learning](https://mlsanity.com/paper/2304.06211/) (2023)
- [Look at What I'm Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos](https://mlsanity.com/paper/2110.10596/) (2021)
- [Learning Hierarchical Image Segmentation For Recognition and By Recognition](https://mlsanity.com/paper/2210.00314/) (2022)
- [Learning Common Rationale to Improve Self-Supervised Representation for Fine-Grained Visual Recognition Problems](https://mlsanity.com/paper/2303.01669/) (2023)
- [Semantic-Aware Fine-Grained Correspondence](https://mlsanity.com/paper/2207.10456/) (2022)
- [Exploring the Coordination of Frequency and Attention in Masked Image Modeling](https://mlsanity.com/paper/2211.15362/) (2022)
- [Correlational Image Modeling for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2303.12670/) (2023)
- [PooDLe: Pooled and dense self-supervised learning from naturalistic videos](https://mlsanity.com/paper/2408.11208/) (2024)
- [Understanding Hyperbolic Metric Learning through Hard Negative Sampling](https://mlsanity.com/paper/2404.15523/) (2024)
- [Exploring Localization for Self-supervised Fine-grained Contrastive Learning](https://mlsanity.com/paper/2106.15788/) (2021)
- [GenView: Enhancing View Quality with Pretrained Generative Model for Self-Supervised Learning](https://mlsanity.com/paper/2403.12003/) (2024)
- [Characterizing and Improving the Robustness of Self-Supervised Learning through Background Augmentations](https://mlsanity.com/paper/2103.12719/) (2021)

## Abstract (arXiv)

Recent advances in self-supervised learning (SSL) have largely closed the gap with supervised ImageNet pretraining. Despite their success these methods have been primarily applied to unlabeled ImageNet images, and show marginal gains when trained on larger sets of uncurated images. We hypothesize that current SSL methods perform best on iconic images, and struggle on complex scene images with many objects. Analyzing contrastive SSL methods shows that they have poor visual grounding and receive poor supervisory signal when trained on scene images. We propose Contrastive Attention-Supervised Tuning(CAST) to overcome these limitations. CAST uses unsupervised saliency maps to intelligently sample crops, and to provide grounding supervision via a Grad-CAM attention loss. Experiments on COCO show that CAST significantly improves the features learned by SSL methods on scene images, and further experiments show that CAST-trained models are more robust to changes in backgrounds.