# AstroLLaMA: Towards Specialized Foundation Models in Astronomy

arXiv: https://arxiv.org/abs/2309.06126 · HTML: https://mlsanity.com/paper/2309.06126/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 18 references that are in the map.
- First posted on arXiv 12 Sep 2023.

## Builds on (most cited first, 12 of 12)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Building astroBERT, a language model for Astronomy & Astrophysics](https://mlsanity.com/paper/2112.00590/) (2021)
- [Harnessing the Power of Adversarial Prompting and Large Language Models for Robust Hypothesis Generation in Astronomy](https://mlsanity.com/paper/2306.11648/) (2023)
- [ADS: The Next Generation Search Platform](https://mlsanity.com/paper/1503.04194/) (2015)
- [Galactic ChitChat: Using Large Language Models to Converse with Astronomy Literature](https://mlsanity.com/paper/2304.05406/) (2023)
- [From Data Processes to Data Products: Knowledge Infrastructures in Astronomy](https://mlsanity.com/paper/2109.01707/) (2021)

## Built on by (most cited first, 30 of 41)

- [A Survey on Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2402.13116/) (2024)
- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [Multiple Physics Pretraining for Physical Surrogate Models](https://mlsanity.com/paper/2310.02994/) (2023)
- [Towards Scientific Intelligence: A Survey of LLM-based Scientific Agents](https://mlsanity.com/paper/2503.24047/) (2025)
- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discovery](https://mlsanity.com/paper/2508.14111/) (2025)
- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [PLLaMa: An Open-source Large Language Model for Plant Science](https://mlsanity.com/paper/2401.01600/) (2024)
- [Large Physics Models: Towards a collaborative approach with Large Language Models and Foundation Models](https://mlsanity.com/paper/2501.05382/) (2025)
- [AstroLLaMA-Chat: Scaling AstroLLaMA with Conversational and Diverse Datasets](https://mlsanity.com/paper/2401.01916/) (2024)
- [Geospatial foundation models for image analysis: evaluating and enhancing NASA-IBM Prithvi's domain adaptability](https://mlsanity.com/paper/2409.00489/) (2024)
- [AstroPT: Scaling Large Observation Models for Astronomy](https://mlsanity.com/paper/2405.14930/) (2024)
- [LoRA-Leak: Membership Inference Attacks Against LoRA Fine-tuned Language Models](https://mlsanity.com/paper/2507.18302/) (2025)
- [Towards a Physics Foundation Model](https://mlsanity.com/paper/2509.13805/) (2025)
- [Delving into the Utilisation of ChatGPT in Scientific Publications in Astronomy](https://mlsanity.com/paper/2406.17324/) (2024)
- [Coarse-to-Fine Highlighting: Reducing Knowledge Hallucination in Large Language Models](https://mlsanity.com/paper/2410.15116/) (2024)
- [AstroMLab 2: AstroLLaMA-2-70B Model and Benchmarking Specialised LLMs for Astronomy](https://mlsanity.com/paper/2409.19750/) (2024)
- [AstroLLaVA: towards the unification of astronomical data and natural language](https://mlsanity.com/paper/2504.08583/) (2025)
- [The Platonic Universe: Do Foundation Models See the Same Sky?](https://mlsanity.com/paper/2509.19453/) (2025)
- [Agentic Exploration of Physics Models](https://mlsanity.com/paper/2509.24978/) (2025)
- [A Short Survey on Small Reasoning Models: Training, Inference, Applications and Research Directions](https://mlsanity.com/paper/2504.09100/) (2025)
- [Physics Steering: Causal Control of Cross-Domain Concepts in a Physics Foundation Model](https://mlsanity.com/paper/2511.20798/) (2025)
- [Designing an Evaluation Framework for Large Language Models in Astronomy Research](https://mlsanity.com/paper/2405.20389/) (2024)
- [Question Rephrasing for Quantifying Uncertainty in Large Language Models: Applications in Molecular Chemistry Tasks](https://mlsanity.com/paper/2408.03732/) (2024)
- [FeynTune: Large Language Models for High-Energy Theory](https://mlsanity.com/paper/2508.03716/) (2025)
- [A Survey of Large Language Models in Discipline-specific Research: Challenges, Methods and Opportunities](https://mlsanity.com/paper/2507.08425/) (2025)
- [From Queries to Criteria: Understanding How Astronomers Evaluate LLMs](https://mlsanity.com/paper/2507.15715/) (2025)
- [ORBIT: Cost-Effective Dataset Curation for Large Language Model Domain Adaptation with an Astronomy Case Study](https://mlsanity.com/paper/2412.14436/) (2024)
- [From Raw Corpora to Domain Benchmarks: Automated Evaluation of LLM Domain Expertise](https://mlsanity.com/paper/2506.07658/) (2025)
- [Discovery of Interpretable Surrogates via Agentic AI: Application to Gravitational Waves](https://mlsanity.com/paper/2605.11280/) (2026)

## Abstract (arXiv)

Large language models excel in many human-language tasks but often falter in highly specialized domains like scholarly astronomy. To bridge this gap, we introduce AstroLLaMA, a 7-billion-parameter model fine-tuned from LLaMA-2 using over 300,000 astronomy abstracts from arXiv. Optimized for traditional causal language modeling, AstroLLaMA achieves a 30% lower perplexity than Llama-2, showing marked domain adaptation. Our model generates more insightful and scientifically relevant text completions and embedding extraction than state-of-the-arts foundation models despite having significantly fewer parameters. AstroLLaMA serves as a robust, domain-specific model with broad fine-tuning potential. Its public release aims to spur astronomy-focused research, including automatic paper summarization and conversational agent development.