mlsanityData to 7 Oct 2026

At First Sight: Zero-Shot Classification of Astronomical Images with Large Multimodal Models

arXiv 2406.17057 ↗First posted 24 Jun 2024astro-ph.IM astro-ph.GA cs.AIcited by 3

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Vision-Language multimodal Models (VLMs) offer the possibility for zero-shot classification in astronomy: i.e. classification via natural language prompts, with no training. We investigate two models, GPT-4o and LLaVA-NeXT, for zero-shot classification of low-surface brightness galaxies and artifacts, as well as morphological classification of galaxies. We show that with natural language prompts these models achieved significant accuracy (above 80 percent typically) without additional training/fine tuning. We discuss areas that require improvement, especially for LLaVA-NeXT, which is an open source model. Our findings aim to motivate the astronomical community to consider VLMs as a powerful tool for both research and pedagogy, with the prospect that future custom-built or fine-tuned models could perform better.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 11 of its 16 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

11 of 16 references.

References · 11

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2023Mistral 7Bcited by 3,065
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  6. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  7. 2024An Introduction to Vision-Language Modelingcited by 175
  8. 2023SoK: Memorization in General-Purpose Large Language Modelscited by 36
  9. 2024PAPERCLIP: Associating Astronomical Observations and Natural Language with Multi-Modal Modelscited by 15
  10. 2022Semi-Supervised Domain Adaptation for Cross-Survey Galaxy Morphology Classification and Anomaly Detectioncited by 4
  11. 2023AstroCLIP: A Cross-Modal Foundation Model for Galaxiescited by 1

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025AstroLLaVA: towards the unification of astronomical data and natural languagecited by 6
  2. 2025Radio Astronomy in the Era of Vision-Language Models: Prompt Sensitivity and Adaptationcited by 2
  3. 2025Semantic search for 100M+ galaxy images using AI-generated captionscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026A General-Purpose VLM Can Teach an Astronomy Foundation Model to Better Recognize Galaxy Morphology
  2. 2024What Do You See? Enhancing Zero-Shot Image Classification with Multimodal Large Language Models
  3. 2024Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions
  4. 2026Beyond Templates: Revisiting Zero-Shot Remote Sensing through Meta-Prompting
  5. 2025AstroMMBench: A Benchmark for Evaluating Multimodal Large Language Models Capabilities in Astronomy
  6. 2026A systematic evaluation of vision-language models for observational astronomical reasoning tasks
  7. 2026AstroAlertBench: Evaluating the Accuracy, Reasoning, and Honesty of Multimodal LLMs in Astronomical Classification
  8. 2023Pushing Boundaries: Exploring Zero Shot Object Classification with Large Multimodal Models