# Multi-Modal Masked Autoencoders for Learning Image-Spectrum Associations for Galaxy Evolution and Cosmology

arXiv: https://arxiv.org/abs/2510.22527 · HTML: https://mlsanity.com/paper/2510.22527/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 31 references that are in the map.
- First posted on arXiv 26 Oct 2025.

## Builds on (most cited first, 11 of 11)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [Deep Multimodal Learning with Missing Modality: A Survey](https://mlsanity.com/paper/2409.07825/) (2024)
- [Self Pre-training with Masked Autoencoders for Medical Image Classification and Segmentation](https://mlsanity.com/paper/2203.05573/) (2022)
- [GalaxiesML: a dataset of galaxy images, photometry, redshifts, and structural parameters for machine learning](https://mlsanity.com/paper/2410.00271/) (2024)
- [Understanding Galaxy Morphology Evolution Through Cosmic Time via Redshift Conditioned Diffusion Models](https://mlsanity.com/paper/2411.18440/) (2024)
- [QuasarNET: Human-level spectral classification and redshifting with Deep Neural Networks](https://mlsanity.com/paper/1808.09955/) (2018)
- [AstroMAE: Redshift Prediction Using a Masked Autoencoder with a Novel Fine-Tuning Architecture](https://mlsanity.com/paper/2409.01825/) (2024)
- [Using different sources of ground truths and transfer learning to improve the generalization of photometric redshift estimation](https://mlsanity.com/paper/2411.18054/) (2024)
- [Category-based Galaxy Image Generation via Diffusion Models](https://mlsanity.com/paper/2506.16255/) (2025)

## Built on by (most cited first, 2 of 2)

- [AquiLLM: An Architecture for Supporting Tacit Knowledge Capture in Research Groups](https://mlsanity.com/paper/2608.08883/) (2026)
- [AquiLLM: Evaluating Faithfulness in Open-Weight RAG-LLM Systems for Scientific Research](https://mlsanity.com/paper/2609.16519/) (2026)

## Abstract (arXiv)

Upcoming surveys will produce billions of galaxy images but comparatively few spectra, motivating models that learn cross-modal representations. We build a dataset of 134,533 galaxy images (HSC-PDR2) and spectra (DESI-DR1) and adapt a Multi-Modal Masked Autoencoder (MMAE) to embed both images and spectra in a shared representation. The MMAE is a transformer-based architecture, which we train by masking 75% of the data and reconstructing missing image and spectral tokens. We use this model to test three applications: spectral and image reconstruction from heavily masked data and redshift regression from images alone. It recovers key physical features, such as galaxy shapes, atomic emission line peaks, and broad continuum slopes, though it struggles with fine image details and line strengths. For redshift regression, the MMAE performs comparably or better than prior multi-modal models in terms of prediction scatter even when missing spectra in testing. These results highlight both the potential and limitations of masked autoencoders in astrophysics and motivate extensions to additional modalities, such as text, for foundation models.