# Self-supervised similarity search for large scientific datasets

arXiv: https://arxiv.org/abs/2110.13151 · HTML: https://mlsanity.com/paper/2110.13151/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 19 references that are in the map.
- First posted on arXiv 25 Oct 2021.

## Builds on (most cited first, 9 of 9)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [CNN Features off-the-shelf: an Astounding Baseline for Recognition](https://mlsanity.com/paper/1403.6382/) (2014)
- [Self-supervised Pretraining of Visual Features in the Wild](https://mlsanity.com/paper/2103.01988/) (2021)
- [Self-Supervised Representation Learning for Astronomical Images](https://mlsanity.com/paper/2012.13083/) (2020)
- [Mining for Strong Gravitational Lenses with Self-supervised Learning](https://mlsanity.com/paper/2110.00023/) (2021)

## Built on by (most cited first, 11 of 11)

- [PAPERCLIP: Associating Astronomical Observations and Natural Language with Multi-Modal Models](https://mlsanity.com/paper/2403.08851/) (2024)
- [Anomalies, Representations, and Self-Supervision](https://mlsanity.com/paper/2301.04660/) (2023)
- [A brief review of contrastive learning applied to astrophysics](https://mlsanity.com/paper/2306.05528/) (2023)
- [Data Compression and Inference in Cosmology with Self-Supervised Machine Learning](https://mlsanity.com/paper/2308.09751/) (2023)
- [Scaling Laws for Galaxy Images](https://mlsanity.com/paper/2404.02973/) (2024)
- [Galaxy Walker: Geometry-aware VLMs For Galaxy-scale Understanding](https://mlsanity.com/paper/2503.18578/) (2025)
- [AstroMAE: Redshift Prediction Using a Masked Autoencoder with a Novel Fine-Tuning Architecture](https://mlsanity.com/paper/2409.01825/) (2024)
- [A review of unsupervised learning in astronomy](https://mlsanity.com/paper/2406.17316/) (2024)
- [AstroCLIP: A Cross-Modal Foundation Model for Galaxies](https://mlsanity.com/paper/2310.03024/) (2023)
- [Semantic search for 100M+ galaxy images using AI-generated captions](https://mlsanity.com/paper/2512.11982/) (2025)
- [A Deep Neural Network Based Reverse Radio Spectrogram Search Algorithm](https://mlsanity.com/paper/2302.13854/) (2023)

## Abstract (arXiv)

We present the use of self-supervised learning to explore and exploit large unlabeled datasets. Focusing on 42 million galaxy images from the latest data release of the Dark Energy Spectroscopic Instrument (DESI) Legacy Imaging Surveys, we first train a self-supervised model to distill low-dimensional representations that are robust to symmetries, uncertainties, and noise in each image. We then use the representations to construct and publicly release an interactive semantic similarity search tool. We demonstrate how our tool can be used to rapidly discover rare objects given only a single example, increase the speed of crowd-sourcing campaigns, and construct and improve training sets for supervised applications. While we focus on images from sky surveys, the technique is straightforward to apply to any scientific dataset of any dimensionality. The similarity search web app can be found at https://github.com/georgestein/galaxy_search