# Steering Self-Supervised Feature Learning Beyond Local Pixel Statistics

arXiv: https://arxiv.org/abs/2004.02331 · HTML: https://mlsanity.com/paper/2004.02331/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Apr 2020.

## Built on by (most cited first, 19 of 19)

- [CutPaste: Self-Supervised Learning for Anomaly Detection and Localization](https://mlsanity.com/paper/2104.04015/) (2021)
- [Transferable Visual Words: Exploiting the Semantics of Anatomical Patterns for Self-supervised Learning](https://mlsanity.com/paper/2102.10680/) (2021)
- [Time-Equivariant Contrastive Video Representation Learning](https://mlsanity.com/paper/2112.03624/) (2021)
- [Audio-Visual Contrastive Learning with Temporal Self-Supervision](https://mlsanity.com/paper/2302.07702/) (2023)
- [Context-Aware Sequence Alignment using 4D Skeletal Augmentation](https://mlsanity.com/paper/2204.12223/) (2022)
- [Joint Self-Supervised Video Alignment and Action Segmentation](https://mlsanity.com/paper/2503.16832/) (2025)
- [Distance-based Hyperspherical Classification for Multi-source Open-Set Domain Adaptation](https://mlsanity.com/paper/2107.02067/) (2021)
- [Self-Supervised Features Improve Open-World Learning](https://mlsanity.com/paper/2102.07848/) (2021)
- [Spatial-Temporal Graph Mamba for Music-Guided Dance Video Synthesis](https://mlsanity.com/paper/2507.06689/) (2025)
- [Procedure Learning via Regularized Gromov-Wasserstein Optimal Transport](https://mlsanity.com/paper/2507.15540/) (2025)
- [Learning by Aligning 2D Skeleton Sequences and Multi-Modality Fusion](https://mlsanity.com/paper/2305.19480/) (2023)
- [Deep learning based domain adaptation for mitochondria segmentation on EM volumes](https://mlsanity.com/paper/2202.10773/) (2022)
- [Semantic Novelty Detection via Relational Reasoning](https://mlsanity.com/paper/2207.08699/) (2022)
- [Evolutionary Augmentation Policy Optimization for Self-supervised Learning](https://mlsanity.com/paper/2303.01584/) (2023)
- [Achieving Domain Robustness in Stereo Matching Networks by Removing Shortcut Learning](https://mlsanity.com/paper/2106.08486/) (2021)
- [Video-ReTime: Learning Temporally Varying Speediness for Time Remapping](https://mlsanity.com/paper/2205.05609/) (2022)
- [Concurrent Discrimination and Alignment for Self-Supervised Feature Learning](https://mlsanity.com/paper/2108.08562/) (2021)
- [Exploring Data Aggregation and Transformations to Generalize across Visual Domains](https://mlsanity.com/paper/2108.09208/) (2021)
- [ScaleNet: An Unsupervised Representation Learning Method for Limited Information](https://mlsanity.com/paper/2310.02386/) (2023)

## Abstract (arXiv)

We introduce a novel principle for self-supervised feature learning based on the discrimination of specific transformations of an image. We argue that the generalization capability of learned features depends on what image neighborhood size is sufficient to discriminate different image transformations: The larger the required neighborhood size and the more global the image statistics that the feature can describe. An accurate description of global image statistics allows to better represent the shape and configuration of objects and their context, which ultimately generalizes better to new tasks such as object classification and detection. This suggests a criterion to choose and design image transformations. Based on this criterion, we introduce a novel image transformation that we call limited context inpainting (LCI). This transformation inpaints an image patch conditioned only on a small rectangular pixel boundary (the limited context). Because of the limited boundary information, the inpainter can learn to match local pixel statistics, but is unlikely to match the global statistics of the image. We claim that the same principle can be used to justify the performance of transformations such as image rotations and warping. Indeed, we demonstrate experimentally that learning to discriminate transformations such as LCI, image warping and rotations, yields features with state of the art generalization capabilities on several datasets such as Pascal VOC, STL-10, CelebA, and ImageNet. Remarkably, our trained features achieve a performance on Places on par with features trained through supervised learning with ImageNet labels.