# Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Prediction

arXiv: https://arxiv.org/abs/2102.10106 · HTML: https://mlsanity.com/paper/2102.10106/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 61 references that are in the map.
- First posted on arXiv 19 Feb 2021.

## Builds on (most cited first, 30 of 43)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Deep Graph Infomax](https://mlsanity.com/paper/1809.10341/) (2018)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [Contrastive Learning with Hard Negative Samples](https://mlsanity.com/paper/2010.04592/) (2020)
- [Deep Graph Contrastive Representation Learning](https://mlsanity.com/paper/2006.04131/) (2020)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [Hard Negative Mixing for Contrastive Learning](https://mlsanity.com/paper/2010.01028/) (2020)
- [The Effectiveness of Data Augmentation in Image Classification using Deep Learning](https://mlsanity.com/paper/1712.04621/) (2017)
- [Keystroke Dynamics as Part of Lifelogging](https://mlsanity.com/paper/2010.10144/) (2020)
- [Whitening for Self-Supervised Representation Learning](https://mlsanity.com/paper/2007.06346/) (2020)
- [What Should Not Be Contrastive in Contrastive Learning](https://mlsanity.com/paper/2008.05659/) (2020)
- [Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction](https://mlsanity.com/paper/1611.09842/) (2016)
- [Markerless tracking of user-defined features with deep learning](https://mlsanity.com/paper/1804.03142/) (2018)
- [Local Aggregation for Unsupervised Learning of Visual Embeddings](https://mlsanity.com/paper/1903.12355/) (2019)
- [Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases](https://mlsanity.com/paper/2007.13916/) (2020)
- [Data augmentation using learned transformations for one-shot medical image segmentation](https://mlsanity.com/paper/1902.09383/) (2019)
- [Uncovering the structure of clinical EEG signals with self-supervised learning](https://mlsanity.com/paper/2007.16104/) (2020)
- [Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learning](https://mlsanity.com/paper/2004.14646/) (2020)

## Built on by (most cited first, 30 of 36)

- [With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2104.14548/) (2021)
- [On Feature Decorrelation in Self-Supervised Learning](https://mlsanity.com/paper/2105.00470/) (2021)
- [Unsupervised Object-Level Representation Learning from Scene Images](https://mlsanity.com/paper/2106.11952/) (2021)
- [A Unified, Scalable Framework for Neural Population Decoding](https://mlsanity.com/paper/2310.16046/) (2023)
- [HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation](https://mlsanity.com/paper/2109.05490/) (2021)
- [Sound Source Localization is All about Cross-Modal Alignment](https://mlsanity.com/paper/2309.10724/) (2023)
- [Mine yOur owN Anatomy: Revisiting Medical Image Segmentation with Extremely Limited Labels](https://mlsanity.com/paper/2209.13476/) (2022)
- [S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captions](https://mlsanity.com/paper/2305.14095/) (2023)
- [The good, the bad and the ugly sides of data augmentation: An implicit spectral regularization perspective](https://mlsanity.com/paper/2210.05021/) (2022)
- [Cross-Trajectory Representation Learning for Zero-Shot Generalization in RL](https://mlsanity.com/paper/2106.02193/) (2021)
- [MIM-Refiner: A Contrastive Learning Boost from Intermediate Pre-Trained Representations](https://mlsanity.com/paper/2402.10093/) (2024)
- [Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity](https://mlsanity.com/paper/2111.02338/) (2021)
- [Seeing the forest and the tree: Building representations of both individual and collective dynamics with transformers](https://mlsanity.com/paper/2206.06131/) (2022)
- [Soft Neighbors are Positive Supporters in Contrastive Visual Representation Learning](https://mlsanity.com/paper/2303.17142/) (2023)
- [CrIBo: Self-Supervised Learning via Cross-Image Object-Level Bootstrapping](https://mlsanity.com/paper/2310.07855/) (2023)
- [Hard Negative Sampling Strategies for Contrastive Representation Learning](https://mlsanity.com/paper/2206.01197/) (2022)
- [Generalizable, real-time neural decoding with hybrid state-space models](https://mlsanity.com/paper/2506.05320/) (2025)
- [Aligning Sight and Sound: Advanced Sound Source Localization Through Audio-Visual Alignment](https://mlsanity.com/paper/2407.13676/) (2024)
- [Understand and Improve Contrastive Learning Methods for Visual Representation: A Review](https://mlsanity.com/paper/2106.03259/) (2021)
- [Constrained Mean Shift Using Distant Yet Related Neighbors for Representation Learning](https://mlsanity.com/paper/2112.04607/) (2021)
- [MTNeuro: A Benchmark for Evaluating Representations of Brain Structure Across Multiple Levels of Abstraction](https://mlsanity.com/paper/2301.00345/) (2023)
- [Augmentation Component Analysis: Modeling Similarity via the Augmentation Overlaps](https://mlsanity.com/paper/2206.00471/) (2022)
- [ProtoCLIP: Prototypical Contrastive Language Image Pretraining](https://mlsanity.com/paper/2206.10996/) (2022)
- [Neural Latent Aligner: Cross-trial Alignment for Learning Representations of Complex, Naturalistic Neural Data](https://mlsanity.com/paper/2308.06443/) (2023)
- [Learning Behavior Representations Through Multi-Timescale Bootstrapping](https://mlsanity.com/paper/2206.07041/) (2022)
- [Relax, it doesn't matter how you get there: A new self-supervised approach for multi-timescale behavior analysis](https://mlsanity.com/paper/2303.08811/) (2023)
- [HistoPerm: A Permutation-Based View Generation Approach for Improving Histopathologic Feature Representation Learning](https://mlsanity.com/paper/2209.06185/) (2022)
- [Synthetic Hard Negative Samples for Contrastive Learning](https://mlsanity.com/paper/2304.02971/) (2023)
- [AutoSciDACT: Automated Scientific Discovery through Contrastive Embedding and Hypothesis Testing](https://mlsanity.com/paper/2510.21935/) (2025)
- [Know Thyself by Knowing Others: Learning Neuron Identity from Population Context](https://mlsanity.com/paper/2512.01199/) (2025)

## Abstract (arXiv)

State-of-the-art methods for self-supervised learning (SSL) build representations by maximizing the similarity between different transformed "views" of a sample. Without sufficient diversity in the transformations used to create views, however, it can be difficult to overcome nuisance variables in the data and build rich representations. This motivates the use of the dataset itself to find similar, yet distinct, samples to serve as views for one another. In this paper, we introduce Mine Your Own vieW (MYOW), a new approach for self-supervised learning that looks within the dataset to define diverse targets for prediction. The idea behind our approach is to actively mine views, finding samples that are neighbors in the representation space of the network, and then predict, from one sample's latent representation, the representation of a nearby sample. After showing the promise of MYOW on benchmarks used in computer vision, we highlight the power of this idea in a novel application in neuroscience where SSL has yet to be applied. When tested on multi-unit neural recordings, we find that MYOW outperforms other self-supervised approaches in all examples (in some cases by more than 10%), and often surpasses the supervised baseline. With MYOW, we show that it is possible to harness the diversity of the data to build rich views and leverage self-supervision in new domains where augmentations are limited or unknown.