# Location-Aware Self-Supervised Transformers for Semantic Segmentation

arXiv: https://arxiv.org/abs/2212.02400 · HTML: https://mlsanity.com/paper/2212.02400/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 84 of its 101 references that are in the map.
- First posted on arXiv 5 Dec 2022.

## Builds on (most cited first, 30 of 84)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)

## Built on by (most cited first, 20 of 20)

- [Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution](https://mlsanity.com/paper/2307.06304/) (2023)
- [MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Features](https://mlsanity.com/paper/2307.12698/) (2023)
- [Towards In-context Scene Understanding](https://mlsanity.com/paper/2306.01667/) (2023)
- [CrIBo: Self-Supervised Learning via Cross-Image Object-Level Bootstrapping](https://mlsanity.com/paper/2310.07855/) (2023)
- [Guided Diffusion from Self-Supervised Diffusion Features](https://mlsanity.com/paper/2312.08825/) (2023)
- [GestFormer: Multiscale Wavelet Pooling Transformer Network for Dynamic Hand Gesture Recognition](https://mlsanity.com/paper/2405.11180/) (2024)
- [Diffusion Masked Pretraining for Dynamic Point Cloud](https://mlsanity.com/paper/2605.03639/) (2026)
- [Self-Supervised Learning for Image Segmentation: A Comprehensive Survey](https://mlsanity.com/paper/2505.13584/) (2025)
- [DIP: Unsupervised Dense In-Context Post-training of Visual Representations](https://mlsanity.com/paper/2506.18463/) (2025)
- [SemanticMIM: Marring Masked Image Modeling with Semantics Compression for General Visual Representation](https://mlsanity.com/paper/2406.10673/) (2024)
- [How PARTs assemble into wholes: Learning the relative composition of images](https://mlsanity.com/paper/2506.03682/) (2025)
- [FlowFeat: Pixel-Dense Embedding of Motion Profiles](https://mlsanity.com/paper/2511.07696/) (2025)
- [Improve Supervised Representation Learning with Masked Image Modeling](https://mlsanity.com/paper/2312.00950/) (2023)
- [Enhancing Semantic Segmentation with Continual Self-Supervised Pre-training](https://mlsanity.com/paper/2509.17816/) (2025)
- [Towards Unsupervised Representation Learning: Learning, Evaluating and Transferring Visual Representations](https://mlsanity.com/paper/2312.00101/) (2023)
- [Learning to Adapt to Position Bias in Vision Transformer Classifiers](https://mlsanity.com/paper/2505.13137/) (2025)
- [SPAR: Self-supervised Placement-Aware Representation Learning for Distributed Sensing](https://mlsanity.com/paper/2505.16936/) (2025)
- [Position Prediction Self-Supervised Learning for Multimodal Satellite Imagery Semantic Segmentation](https://mlsanity.com/paper/2506.06852/) (2025)
- [Stable Single-Pixel Contrastive Learning for Semantic and Geometric Tasks](https://mlsanity.com/paper/2512.04970/) (2025)
- [Featurising Pixels from Dynamic 3D Scenes with Linear In-Context Learners](https://mlsanity.com/paper/2604.26488/) (2026)

## Abstract (arXiv)

Pixel-level labels are particularly expensive to acquire. Hence, pretraining is a critical step to improve models on a task like semantic segmentation. However, prominent algorithms for pretraining neural networks use image-level objectives, e.g. image classification, image-text alignment a la CLIP, or self-supervised contrastive learning. These objectives do not model spatial information, which might be sub-optimal when finetuning on downstream tasks with spatial reasoning. In this work, we pretrain network with a location-aware (LOCA) self-supervised method which fosters the emergence of strong dense features. Specifically, we use both a patch-level clustering scheme to mine dense pseudo-labels and a relative location prediction task to encourage learning about object parts and their spatial arrangements. Our experiments show that LOCA pretraining leads to representations that transfer competitively to challenging and diverse semantic segmentation datasets.