# GeoGalactica: A Scientific Large Language Model in Geoscience

arXiv: https://arxiv.org/abs/2401.00434 · HTML: https://mlsanity.com/paper/2401.00434/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 68 references that are in the map.
- First posted on arXiv 31 Dec 2023.

## Builds on (most cited first, 21 of 21)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Automatic Chain of Thought Prompting in Large Language Models](https://mlsanity.com/paper/2210.03493/) (2022)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [LEGAL-BERT: The Muppets straight out of Law School](https://mlsanity.com/paper/2010.02559/) (2020)
- [K2: A Foundation Language Model for Geoscience Knowledge Understanding and Utilization](https://mlsanity.com/paper/2306.05064/) (2023)
- [An Iterative Optimizing Framework for Radiology Report Summarization with ChatGPT](https://mlsanity.com/paper/2304.08448/) (2023)
- [Radiology-GPT: A Large Language Model for Radiology](https://mlsanity.com/paper/2306.08666/) (2023)
- [GeoGPT: Understanding and Processing Geospatial Tasks through An Autonomous GPT](https://mlsanity.com/paper/2307.07930/) (2023)
- [DARWIN Series: Domain Specific Large Language Models for Natural Science](https://mlsanity.com/paper/2308.13565/) (2023)
- [Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data](https://mlsanity.com/paper/2304.01196/) (2023)
- [When Geoscience Meets Foundation Models: Towards General Geoscience Artificial Intelligence System](https://mlsanity.com/paper/2309.06799/) (2023)
- [Soft Prompt Tuning for Augmenting Dense Retrieval with Large Language Models](https://mlsanity.com/paper/2307.08303/) (2023)
- [reStructured Pre-training](https://mlsanity.com/paper/2206.11147/) (2022)
- [DeepShovel: An Online Collaborative Platform for Data Extraction in Geoscience Literature with AI Assistance](https://mlsanity.com/paper/2202.10163/) (2022)
- [PK-Chat: Pointer Network Guided Knowledge Driven Generative Dialogue Model](https://mlsanity.com/paper/2304.00592/) (2023)

## Built on by (most cited first, 30 of 33)

- [A Survey on Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2402.13116/) (2024)
- [LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model](https://mlsanity.com/paper/2402.02544/) (2024)
- [DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning](https://mlsanity.com/paper/2402.17453/) (2024)
- [Knowledge Mechanisms in Large Language Models: A Survey and Perspective](https://mlsanity.com/paper/2407.15017/) (2024)
- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [Scientists' First Exam: Probing Cognitive Abilities of MLLM via Perception, Understanding, and Reasoning](https://mlsanity.com/paper/2506.10521/) (2025)
- [NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens](https://mlsanity.com/paper/2403.12766/) (2024)
- [PLLaMa: An Open-source Large Language Model for Plant Science](https://mlsanity.com/paper/2401.01600/) (2024)
- [An LLM Agent for Automatic Geospatial Data Analysis](https://mlsanity.com/paper/2410.18792/) (2024)
- [UrbanKGent: A Unified Large Language Model Agent Framework for Urban Knowledge Graph Construction](https://mlsanity.com/paper/2402.06861/) (2024)
- [Accelerating Earth Science Discovery via Multi-Agent LLM Systems](https://mlsanity.com/paper/2503.05854/) (2025)
- [SciDFM: A Large Language Model with Mixture-of-Experts for Science](https://mlsanity.com/paper/2409.18412/) (2024)
- [PEACE: Empowering Geologic Map Holistic Understanding with MLLMs](https://mlsanity.com/paper/2501.06184/) (2025)
- [PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computing](https://mlsanity.com/paper/2503.12167/) (2025)
- [Foundation Models for Environmental Science: A Survey of Emerging Frontiers](https://mlsanity.com/paper/2504.04280/) (2025)
- [Innovator: Scientific Continued Pretraining with Fine-grained MoE Upcycling](https://mlsanity.com/paper/2507.18671/) (2025)
- [Language Models Do Not Embed Numbers Continuously](https://mlsanity.com/paper/2510.08009/) (2025)
- [A Hierarchical Multi-Agent System for Autonomous Discovery in Geoscientific Data Archives](https://mlsanity.com/paper/2602.21351/) (2026)
- [Thoth: Mid-Training Bridges LLMs to Time Series Understanding](https://mlsanity.com/paper/2603.01042/) (2026)
- [A Survey of Foundation Models for Environmental Science](https://mlsanity.com/paper/2503.03142/) (2025)
- [A Vision for Geo-Temporal Deep Research Systems: Towards Comprehensive, Transparent, and Reproducible Geo-Temporal Information Synthesis](https://mlsanity.com/paper/2506.14345/) (2025)
- [Artificial Intelligence for Atmospheric Sciences: A Research Roadmap](https://mlsanity.com/paper/2506.16281/) (2025)
- [KALE-LM-Chem: Vision and Practice Toward an AI Brain for Chemistry](https://mlsanity.com/paper/2409.18695/) (2024)
- [METEOR: Evolutionary Journey of Large Language Models from Guidance to Self-Growth](https://mlsanity.com/paper/2411.11933/) (2024)
- [Using LLMs for Analyzing AIS Data](https://mlsanity.com/paper/2504.07557/) (2025)
- [Domain adaptation of large language models for geotechnical applications](https://mlsanity.com/paper/2507.05613/) (2025)
- [Geolog-IA: Conversational System for Academic Theses](https://mlsanity.com/paper/2510.02653/) (2025)
- [QueryPlot: Generating Geological Evidence Layers using Natural Language Queries for Mineral Exploration](https://mlsanity.com/paper/2602.17784/) (2026)
- [GeoMind: An Agentic Workflow for Lithology Classification with Reasoned Tool Invocation](https://mlsanity.com/paper/2604.21501/) (2026)
- [Earth Science Foundation Models: From Perception to Reasoning and Discovery](https://mlsanity.com/paper/2605.12542/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have achieved huge success for their general knowledge and ability to solve a wide spectrum of tasks in natural language processing (NLP). Due to their impressive abilities, LLMs have shed light on potential inter-discipline applications to foster scientific discoveries of a specific domain by using artificial intelligence (AI for science, AI4S). In the meantime, utilizing NLP techniques in geoscience research and practice is wide and convoluted, contributing from knowledge extraction and document classification to question answering and knowledge discovery. In this work, we take the initial step to leverage LLM for science, through a rather straightforward approach. We try to specialize an LLM into geoscience, by further pre-training the model with a vast amount of texts in geoscience, as well as supervised fine-tuning (SFT) the resulting model with our custom collected instruction tuning dataset. These efforts result in a model GeoGalactica consisting of 30 billion parameters. To our best knowledge, it is the largest language model for the geoscience domain. More specifically, GeoGalactica is from further pre-training of Galactica. We train GeoGalactica over a geoscience-related text corpus containing 65 billion tokens, preserving as the largest geoscience-specific text corpus. Then we fine-tune the model with 1 million pairs of instruction-tuning data consisting of questions that demand professional geoscience knowledge to answer. In this technical report, we will illustrate in detail all aspects of GeoGalactica, including data collection, data cleaning, base model selection, pre-training, SFT, and evaluation. We open-source our data curation tools and the checkpoints of GeoGalactica during the first 3/4 of pre-training.