# AstroCLIP: A Cross-Modal Foundation Model for Galaxies

arXiv: https://arxiv.org/abs/2310.03024 · HTML: https://mlsanity.com/paper/2310.03024/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 36 references that are in the map.
- First posted on arXiv 4 Oct 2023.

## Builds on (most cited first, 6 of 6)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Self-supervised similarity search for large scientific datasets](https://mlsanity.com/paper/2110.13151/) (2021)
- [Towards Galaxy Foundation Models with Hybrid Contrastive Learning](https://mlsanity.com/paper/2206.11927/) (2022)
- [Predicting galaxy spectra from images with hybrid convolutional neural networks](https://mlsanity.com/paper/2009.12318/) (2020)
- [Estimating Galactic Distances From Images Using Self-supervised Representation Learning](https://mlsanity.com/paper/2101.04293/) (2021)

## Built on by (most cited first, 1 of 1)

- [At First Sight: Zero-Shot Classification of Astronomical Images with Large Multimodal Models](https://mlsanity.com/paper/2406.17057/) (2024)

## Abstract (arXiv)

We present AstroCLIP, a single, versatile model that can embed both galaxy images and spectra into a shared, physically meaningful latent space. These embeddings can then be used - without any model fine-tuning - for a variety of downstream tasks including (1) accurate in-modality and cross-modality semantic similarity search, (2) photometric redshift estimation, (3) galaxy property estimation from both images and spectra, and (4) morphology classification. Our approach to implementing AstroCLIP consists of two parts. First, we embed galaxy images and spectra separately by pretraining separate transformer-based image and spectrum encoders in self-supervised settings. We then align the encoders using a contrastive loss. We apply our method to spectra from the Dark Energy Spectroscopic Instrument and images from its corresponding Legacy Imaging Survey. Overall, we find remarkable performance on all downstream tasks, even relative to supervised baselines. For example, for a task like photometric redshift prediction, we find similar performance to a specifically-trained ResNet18, and for additional tasks like physical property estimation (stellar mass, age, metallicity, and sSFR), we beat this supervised baseline by 19% in terms of R2. We also compare our results to a state-of-the-art self-supervised single-modal model for galaxy images, and find that our approach outperforms this benchmark by roughly a factor of two on photometric redshift estimation and physical property prediction in terms of R2, while remaining roughly in-line in terms of morphology classification. Ultimately, our approach represents the first cross-modal self-supervised model for galaxies, and the first self-supervised transformer-based architectures for galaxy images and spectra.