# AstroLLaVA: towards the unification of astronomical data and natural language

arXiv: https://arxiv.org/abs/2504.08583 · HTML: https://mlsanity.com/paper/2504.08583/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 16 of its 30 references that are in the map.
- First posted on arXiv 11 Apr 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 16 of 16)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Physically Grounded Vision-Language Models for Robotic Manipulation](https://mlsanity.com/paper/2309.02561/) (2023)
- [Galaxy Zoo DECaLS: Detailed Visual Morphology Measurements from Volunteers and Deep Learning for 314,000 Galaxies](https://mlsanity.com/paper/2102.08414/) (2021)
- [AstroLLaMA: Towards Specialized Foundation Models in Astronomy](https://mlsanity.com/paper/2309.06126/) (2023)
- [On Large Visual Language Models for Medical Imaging Analysis: An Empirical Study](https://mlsanity.com/paper/2402.14162/) (2024)
- [PAPERCLIP: Associating Astronomical Observations and Natural Language with Multi-Modal Models](https://mlsanity.com/paper/2403.08851/) (2024)
- [EleutherAI: Going Beyond "Open Science" to "Science in the Open"](https://mlsanity.com/paper/2210.06413/) (2022)
- [AstroLLaMA-Chat: Scaling AstroLLaMA with Conversational and Diverse Datasets](https://mlsanity.com/paper/2401.01916/) (2024)
- [At First Sight: Zero-Shot Classification of Astronomical Images with Large Multimodal Models](https://mlsanity.com/paper/2406.17057/) (2024)
- [A Versatile Framework for Analyzing Galaxy Image Data by Implanting Human-in-the-loop on a Large Vision Model](https://mlsanity.com/paper/2405.10890/) (2024)
- [herakoi: a sonification experiment for astronomical data](https://mlsanity.com/paper/2412.09152/) (2024)

## Built on by (most cited first, 6 of 6)

- [From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discovery](https://mlsanity.com/paper/2508.14111/) (2025)
- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini](https://mlsanity.com/paper/2605.27295/) (2026)
- [The Platonic Universe: Do Foundation Models See the Same Sky?](https://mlsanity.com/paper/2509.19453/) (2025)
- [Radio Astronomy in the Era of Vision-Language Models: Prompt Sensitivity and Adaptation](https://mlsanity.com/paper/2509.02615/) (2025)
- [Generalist Vision-Language Models for Fast Radio Burst detection: a zero-shot benchmark against a specialized detector](https://mlsanity.com/paper/2607.07382/) (2026)

## Abstract (arXiv)

We present AstroLLaVA, a vision language model for astronomy that enables interaction with astronomical imagery through natural dialogue. By fine-tuning the LLaVA model on a diverse dataset of ~30k images with captions and question-answer pairs sourced from NASA's `Astronomy Picture of the Day', the European Southern Observatory, and the NASA/ESA Hubble Space Telescope, we create a model capable of answering open-ended questions about astronomical concepts depicted visually. Our two-stage fine-tuning process adapts the model to both image captioning and visual question answering in the astronomy domain. We demonstrate AstroLLaVA's performance on an astronomical visual question answering benchmark and release the model weights, code, and training set to encourage further open source work in this space. Finally, we suggest a roadmap towards general astronomical data alignment with pre-trained language models, and provide an open space for collaboration towards this end for interested researchers.