# K2: A Foundation Language Model for Geoscience Knowledge Understanding and Utilization

arXiv: https://arxiv.org/abs/2306.05064 · HTML: https://mlsanity.com/paper/2306.05064/

## Key facts

- Cited by 69 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 59 references that are in the map.
- First posted on arXiv 8 Jun 2023.

## Builds on (most cited first, 30 of 31)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [GPTScore: Evaluate as You Desire](https://mlsanity.com/paper/2302.04166/) (2023)
- [On the Opportunities and Challenges of Foundation Models for Geospatial Artificial Intelligence](https://mlsanity.com/paper/2304.06798/) (2023)
- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [MatSciBERT: A Materials Domain Language Model for Text Mining and Information Extraction](https://mlsanity.com/paper/2109.15290/) (2021)
- [BioMegatron: Larger Biomedical Domain Language Model](https://mlsanity.com/paper/2010.06060/) (2020)
- [A Review of Location Encoding for GeoAI: Methods and Applications](https://mlsanity.com/paper/2111.04006/) (2021)
- [ERNIE-GeoL: A Geography-and-Language Pre-trained Model and its Applications in Baidu Maps](https://mlsanity.com/paper/2203.09127/) (2022)
- [MedGPT: Medical Concept Prediction from Clinical Narratives](https://mlsanity.com/paper/2107.03134/) (2021)
- [Exploring and Verbalizing Academic Ideas by Concept Co-occurrence](https://mlsanity.com/paper/2306.02282/) (2023)
- [DeepShovel: An Online Collaborative Platform for Data Extraction in Geoscience Literature with AI Assistance](https://mlsanity.com/paper/2202.10163/) (2022)

## Built on by (most cited first, 30 of 69)

- [A Survey on Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2402.13116/) (2024)
- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning](https://mlsanity.com/paper/2402.17453/) (2024)
- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Models](https://mlsanity.com/paper/2501.13958/) (2025)
- [Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application](https://mlsanity.com/paper/2407.01885/) (2024)
- [Developing ChemDFM as a large language foundation model for chemistry](https://mlsanity.com/paper/2401.14818/) (2024)
- [Knowledge Mechanisms in Large Language Models: A Survey and Perspective](https://mlsanity.com/paper/2407.15017/) (2024)
- [GeoGalactica: A Scientific Large Language Model in Geoscience](https://mlsanity.com/paper/2401.00434/) (2023)
- [CityGPT: Empowering Urban Spatial Cognition of Large Language Models](https://mlsanity.com/paper/2406.13948/) (2024)
- [Towards Urban General Intelligence: A Review and Outlook of Urban Foundation Models](https://mlsanity.com/paper/2402.01749/) (2024)
- [An LLM Agent for Automatic Geospatial Data Analysis](https://mlsanity.com/paper/2410.18792/) (2024)
- [Where to Move Next: Zero-shot Generalization of LLMs for Next POI Recommendation](https://mlsanity.com/paper/2404.01855/) (2024)
- [Geolocation Representation from Large Language Models are Generic Enhancers for Spatio-Temporal Learning](https://mlsanity.com/paper/2408.12116/) (2024)
- [AI Sees Your Location, But With A Bias Toward The Wealthy World](https://mlsanity.com/paper/2502.11163/) (2025)
- [When Geoscience Meets Foundation Models: Towards General Geoscience Artificial Intelligence System](https://mlsanity.com/paper/2309.06799/) (2023)
- [GeoLocator: a location-integrated large multimodal model for inferring geo-privacy](https://mlsanity.com/paper/2311.13018/) (2023)
- [Accelerating Earth Science Discovery via Multi-Agent LLM Systems](https://mlsanity.com/paper/2503.05854/) (2025)
- [Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement](https://mlsanity.com/paper/2402.06700/) (2024)
- [From Generalist to Specialist: A Survey of Large Language Models for Chemistry](https://mlsanity.com/paper/2412.19994/) (2024)
- [Self-Supervised Representation Learning for Geospatial Objects: A Survey](https://mlsanity.com/paper/2408.12133/) (2024)
- [A Survey of Pre-trained Language Models for Processing Scientific Text](https://mlsanity.com/paper/2401.17824/) (2024)
- [MING-MOE: Enhancing Medical Multi-Task Learning in Large Language Models with Sparse Mixture of Low-Rank Adapter Experts](https://mlsanity.com/paper/2404.09027/) (2024)
- [PEACE: Empowering Geologic Map Holistic Understanding with MLLMs](https://mlsanity.com/paper/2501.06184/) (2025)
- [EarthSE: A Benchmark for Evaluating Earth Scientific Exploration Capability of LLMs](https://mlsanity.com/paper/2505.17139/) (2025)
- [ScIRGen: Synthesize Realistic and Large-Scale RAG Dataset for Scientific Research](https://mlsanity.com/paper/2506.11117/) (2025)
- [MoFM: A Large-Scale Human Motion Foundation Model](https://mlsanity.com/paper/2502.05432/) (2025)
- [GeoAgentBench: A Dynamic Execution Benchmark for Tool-Augmented Agents in Spatial Analysis](https://mlsanity.com/paper/2604.13888/) (2026)
- [Future-proofing geotechnics workflows: accelerating problem-solving with large language models](https://mlsanity.com/paper/2312.12411/) (2023)
- [Automatically Generating Numerous Context-Driven SFT Data for LLMs across Diverse Granularity](https://mlsanity.com/paper/2405.16579/) (2024)

## Abstract (arXiv)

Large language models (LLMs) have achieved great success in general domains of natural language processing. In this paper, we bring LLMs to the realm of geoscience with the objective of advancing research and applications in this field. To this end, we present the first-ever LLM in geoscience, K2, alongside a suite of resources developed to further promote LLM research within geoscience. For instance, we have curated the first geoscience instruction tuning dataset, GeoSignal, which aims to align LLM responses to geoscience-related user queries. Additionally, we have established the first geoscience benchmark, GeoBench, to evaluate LLMs in the context of geoscience. In this work, we experiment with a complete recipe to adapt a pre-trained general-domain LLM to the geoscience domain. Specifically, we further train the LLaMA-7B model on 5.5B tokens of geoscience text corpus, including over 1 million pieces of geoscience literature, and utilize GeoSignal's supervised data to fine-tune the model. Moreover, we share a protocol that can efficiently gather domain-specific data and construct domain-supervised data, even in situations where manpower is scarce. Meanwhile, we equip K2 with the abilities of using tools to be a naive geoscience aide. Experiments conducted on the GeoBench demonstrate the effectiveness of our approach and datasets on geoscience knowledge understanding and utilization.We open-source all the training data and K2 model checkpoints at https://github.com/davendw49/k2.