mlsanityData to 7 Oct 2026

AstroLLaVA: towards the unification of astronomical data and natural language

arXiv 2504.08583 ↗First posted 11 Apr 2025astro-ph.IM cs.LGcited by 6

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

We present AstroLLaVA, a vision language model for astronomy that enables interaction with astronomical imagery through natural dialogue. By fine-tuning the LLaVA model on a diverse dataset of ~30k images with captions and question-answer pairs sourced from NASA's `Astronomy Picture of the Day', the European Southern Observatory, and the NASA/ESA Hubble Space Telescope, we create a model capable of answering open-ended questions about astronomical concepts depicted visually. Our two-stage fine-tuning process adapts the model to both image captioning and visual question answering in the astronomy domain. We demonstrate AstroLLaVA's performance on an astronomical visual question answering benchmark and release the model weights, code, and training set to encourage further open source work in this space. Finally, we suggest a roadmap towards general astronomical data alignment with pre-trained language models, and provide an open space for collaboration towards this end for interested researchers.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 30 references that are in the map. First posted on arXiv 11 Apr 2025.

Builds on

16 of 30 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024At First Sight: Zero-Shot Classification of Astronomical Images with Large Multimodal Models24 Jun
  2. 2023Physically Grounded Vision-Language Models for Robotic Manipulation5 Sep

Other references · 14

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  7. 2023AstroLLaMA: Towards Specialized Foundation Models in Astronomycited by 41
  8. 2021Galaxy Zoo DECaLS: Detailed Visual Morphology Measurements from Volunteers and Deep Learning for 314,000 Galaxiescited by 40
  9. 2024On Large Visual Language Models for Medical Imaging Analysis: An Empirical Studycited by 28
  10. 2024PAPERCLIP: Associating Astronomical Observations and Natural Language with Multi-Modal Modelscited by 15
  11. 2022EleutherAI: Going Beyond "Open Science" to "Science in the Open"cited by 13
  12. 2024AstroLLaMA-Chat: Scaling AstroLLaMA with Conversational and Diverse Datasetscited by 13
2 more
  1. 2024A Versatile Framework for Analyzing Galaxy Image Data by Implanting Human-in-the-loop on a Large Vision Modelcited by 2
  2. 2024herakoi: a sonification experiment for astronomical datacited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 74
  2. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  3. 2025The Platonic Universe: Do Foundation Models See the Same Sky?cited by 5
  4. 2026Gemini Embedding 2: A Native Multimodal Embedding Model from Geminicited by 5
  5. 2025Radio Astronomy in the Era of Vision-Language Models: Prompt Sensitivity and Adaptationcited by 2
  6. 2026Generalist Vision-Language Models for Fast Radio Burst detection: a zero-shot benchmark against a specialized detectorcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AstroMMBench: A Benchmark for Evaluating Multimodal Large Language Models Capabilities in Astronomy
  2. 2026A systematic evaluation of vision-language models for observational astronomical reasoning tasks
  3. 2025AstroMLab 4: Benchmark-Topping Performance in Astronomy Q&A with a 70B-Parameter Domain-Specialized Reasoning Model
  4. 2025VeriSciQA: An Auto-Verified Dataset for Scientific Visual Question Answering
  5. 2026LLaVA-LE: Large Language-and-Vision Assistant for Lunar Exploration
  6. 2025Semantic search for 100M+ galaxy images using AI-generated captions
  7. 2026Rethinking Domain Specialization for Open-Ended Scientific Reasoning in Astronomy Language Models
  8. 2024Large Vision-Language Models for Remote Sensing Visual Question Answering