mlsanityData to 7 Oct 2026

AstroLLaMA-Chat: Scaling AstroLLaMA with Conversational and Diverse Datasets

arXiv 2401.01916 ↗First posted 3 Jan 2024astro-ph.IM astro-ph.CO astro-ph.GAcited by 13

Not on the map. Why?

Abstract (arXiv)

We explore the potential of enhancing LLM performance in astronomy-focused question-answering through targeted, continual pre-training. By employing a compact 7B-parameter LLaMA-2 model and focusing exclusively on a curated set of astronomy corpora -- comprising abstracts, introductions, and conclusions -- we achieve notable improvements in specialized topic comprehension. While general LLMs like GPT-4 excel in broader question-answering scenarios due to superior reasoning capabilities, our findings suggest that continual pre-training with limited resources can still enhance model performance on specialized topics. Additionally, we present an extension of AstroLLaMA: the fine-tuning of the 7B LLaMA model on a domain-specific conversational dataset, culminating in the release of the chat-enabled AstroLLaMA for community use. Comprehensive quantitative benchmarking is currently in progress and will be detailed in an upcoming full paper. The model, AstroLLaMA-Chat, is now available at https://huggingface.co/universeTBD, providing the first open-source conversational AI tool tailored for the astronomy community.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 10 of its 13 references that are in the map. First posted on arXiv 3 Jan 2024.

Builds on

10 of 13 references.

References · 10

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  6. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  7. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  8. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  9. 2023LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Modelscited by 58
  10. 2023AstroLLaMA: Towards Specialized Foundation Models in Astronomycited by 41

Built on by

Cited by year

Citing papers per year: 2024: 7; 2025: 6.

13 papers.

Sort:

By area:

  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  2. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  3. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  4. 2024AstroPT: Scaling Large Observation Models for Astronomycited by 9
  5. 2024Delving into the Utilisation of ChatGPT in Scientific Publications in Astronomycited by 7
  6. 2024AstroMLab 2: AstroLLaMA-2-70B Model and Benchmarking Specialised LLMs for Astronomycited by 6
  7. 2025AstroLLaVA: towards the unification of astronomical data and natural languagecited by 6
  8. 2025The Platonic Universe: Do Foundation Models See the Same Sky?cited by 5
  9. 2024Designing an Evaluation Framework for Large Language Models in Astronomy Researchcited by 3
  10. 2025A Survey of Large Language Models in Discipline-specific Research: Challenges, Methods and Opportunitiescited by 2
  11. 2025From Queries to Criteria: Understanding How Astronomers Evaluate LLMscited by 2
  12. 2025FeynTune: Large Language Models for High-Energy Theorycited by 2
1 more
  1. 2025AstroReview: An LLM-driven Multi-Agent Framework for Telescope Proposal Peer Review and Refinementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AstroMLab 4: Benchmark-Topping Performance in Astronomy Q&A with a 70B-Parameter Domain-Specialized Reasoning Model
  2. 2025Large Language Models Achieve Gold Medal Performance at the International Olympiad on Astronomy & Astrophysics (IOAA)
  3. 2026Taramandal-GPT: Enhancing Astrodynamics Problem-Solving with Knowledge Retrieval and Structured Thinking
  4. 2026Rethinking Domain Specialization for Open-Ended Scientific Reasoning in Astronomy Language Models
  5. 2026AstroRAG -- A Pagerank-Based Retrieval-Augmented Generation Pipeline for Question Answering in Astronomy
  6. 2025AstroMMBench: A Benchmark for Evaluating Multimodal Large Language Models Capabilities in Astronomy
  7. 2026AstroAlertBench: Evaluating the Accuracy, Reasoning, and Honesty of Multimodal LLMs in Astronomical Classification
  8. 2023T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Language Model Signals for Science Question Answering