# GalaxAlign: Mimicking Citizen Scientists' Multimodal Guidance for Galaxy Morphology Analysis

arXiv: https://arxiv.org/abs/2411.19475 · HTML: https://mlsanity.com/paper/2411.19475/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 36 references that are in the map.
- First posted on arXiv 29 Nov 2024.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023)
- [GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localization](https://mlsanity.com/paper/2309.16020/) (2023)
- [Galaxy Zoo DECaLS: Detailed Visual Morphology Measurements from Volunteers and Deep Learning for 314,000 Galaxies](https://mlsanity.com/paper/2102.08414/) (2021)
- [Galaxy Zoo: Probabilistic Morphology through Bayesian CNNs and Active Learning](https://mlsanity.com/paper/1905.07424/) (2019)
- [Practical Galaxy Morphology Tools from Deep Supervised Representation Learning](https://mlsanity.com/paper/2110.12735/) (2021)
- [PAPERCLIP: Associating Astronomical Observations and Natural Language with Multi-Modal Models](https://mlsanity.com/paper/2403.08851/) (2024)
- [CosmoCLIP: Generalizing Large Vision-Language Models for Astronomical Imaging](https://mlsanity.com/paper/2407.07315/) (2024)
- [A New Task: Deriving Semantic Class Targets for the Physical Sciences](https://mlsanity.com/paper/2210.14760/) (2022)
- [CzSL: Learning from citizen science, experts and unlabelled data in astronomical image classification](https://mlsanity.com/paper/2302.00366/) (2023)

## Built on by (most cited first, 4 of 4)

- [Are vision language models robust to uncertain inputs?](https://mlsanity.com/paper/2505.11804/) (2025)
- [D-GAP: Improving Out-of-Domain Robustness via Dataset-Agnostic and Gradient-Guided Augmentation in Frequency and Pixel Spaces](https://mlsanity.com/paper/2511.11286/) (2025)
- [Improving Radio Interferometry Imaging by Explicitly Modeling Cross-Domain Consistency in Reconstruction](https://mlsanity.com/paper/2604.16794/) (2026)
- [Estimating Uncertainty in Galaxy Morphology Classification](https://mlsanity.com/paper/2608.08398/) (2026)

## Abstract (arXiv)

Galaxy morphology analysis involves studying galaxies based on their shapes and structures. For such studies, fundamental tasks include identifying and classifying galaxies in astronomical images, as well as retrieving visually or structurally similar galaxies through similarity search. Existing methods either directly train domain-specific foundation models on large, annotated datasets or fine-tune vision foundation models on a smaller set of images. The former is effective but costly, while the latter is more resource-efficient but often yields lower accuracy. To address these challenges, we introduce GalaxAlign, a multimodal approach inspired by how citizen scientists identify galaxies in astronomical images by following textual descriptions and matching schematic symbols. Specifically, GalaxAlign employs a tri-modal alignment framework to align three types of data during fine-tuning: (1) schematic symbols representing galaxy shapes and structures, (2) textual labels for these symbols, and (3) galaxy images. By incorporating multimodal instructions, GalaxAlign eliminates the need for expensive pretraining and enhances the effectiveness of fine-tuning. Experiments on galaxy classification and similarity search demonstrate that our method effectively fine-tunes general pre-trained models for astronomical tasks by incorporating domain-specific multi-modal knowledge. Code is available at https://github.com/RapidsAtHKUST/GalaxAlign.