# AstroLLaMA-Chat: Scaling AstroLLaMA with Conversational and Diverse Datasets

arXiv: https://arxiv.org/abs/2401.01916 · HTML: https://mlsanity.com/paper/2401.01916/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 13 references that are in the map.
- First posted on arXiv 3 Jan 2024.

## Builds on (most cited first, 10 of 10)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [Orca: Progressive Learning from Complex Explanation Traces of GPT-4](https://mlsanity.com/paper/2306.02707/) (2023)
- [LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models](https://mlsanity.com/paper/2306.12420/) (2023)
- [AstroLLaMA: Towards Specialized Foundation Models in Astronomy](https://mlsanity.com/paper/2309.06126/) (2023)

## Built on by (most cited first, 13 of 13)

- [A Survey on Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2402.13116/) (2024)
- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [AstroPT: Scaling Large Observation Models for Astronomy](https://mlsanity.com/paper/2405.14930/) (2024)
- [Delving into the Utilisation of ChatGPT in Scientific Publications in Astronomy](https://mlsanity.com/paper/2406.17324/) (2024)
- [AstroMLab 2: AstroLLaMA-2-70B Model and Benchmarking Specialised LLMs for Astronomy](https://mlsanity.com/paper/2409.19750/) (2024)
- [AstroLLaVA: towards the unification of astronomical data and natural language](https://mlsanity.com/paper/2504.08583/) (2025)
- [The Platonic Universe: Do Foundation Models See the Same Sky?](https://mlsanity.com/paper/2509.19453/) (2025)
- [Designing an Evaluation Framework for Large Language Models in Astronomy Research](https://mlsanity.com/paper/2405.20389/) (2024)
- [FeynTune: Large Language Models for High-Energy Theory](https://mlsanity.com/paper/2508.03716/) (2025)
- [A Survey of Large Language Models in Discipline-specific Research: Challenges, Methods and Opportunities](https://mlsanity.com/paper/2507.08425/) (2025)
- [From Queries to Criteria: Understanding How Astronomers Evaluate LLMs](https://mlsanity.com/paper/2507.15715/) (2025)
- [AstroReview: An LLM-driven Multi-Agent Framework for Telescope Proposal Peer Review and Refinement](https://mlsanity.com/paper/2512.24754/) (2025)

## Abstract (arXiv)

We explore the potential of enhancing LLM performance in astronomy-focused question-answering through targeted, continual pre-training. By employing a compact 7B-parameter LLaMA-2 model and focusing exclusively on a curated set of astronomy corpora -- comprising abstracts, introductions, and conclusions -- we achieve notable improvements in specialized topic comprehension. While general LLMs like GPT-4 excel in broader question-answering scenarios due to superior reasoning capabilities, our findings suggest that continual pre-training with limited resources can still enhance model performance on specialized topics. Additionally, we present an extension of AstroLLaMA: the fine-tuning of the 7B LLaMA model on a domain-specific conversational dataset, culminating in the release of the chat-enabled AstroLLaMA for community use. Comprehensive quantitative benchmarking is currently in progress and will be detailed in an upcoming full paper. The model, AstroLLaMA-Chat, is now available at https://huggingface.co/universeTBD, providing the first open-source conversational AI tool tailored for the astronomy community.