# ClimateGPT: Towards AI Synthesizing Interdisciplinary Research on Climate Change

arXiv: https://arxiv.org/abs/2401.09646 · HTML: https://mlsanity.com/paper/2401.09646/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jan 2024.

## Built on by (most cited first, 30 of 47)

- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [LLM Post-Training: A Deep Dive into Reasoning Large Language Models](https://mlsanity.com/paper/2502.21321/) (2025)
- [Towards Scientific Intelligence: A Survey of LLM-based Scientific Agents](https://mlsanity.com/paper/2503.24047/) (2025)
- [From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discovery](https://mlsanity.com/paper/2508.14111/) (2025)
- [Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers](https://mlsanity.com/paper/2404.04925/) (2024)
- [On the Foundations of Earth and Climate Foundation Models](https://mlsanity.com/paper/2405.04285/) (2024)
- [Learning to Generate Instruction Tuning Datasets for Zero-Shot Task Adaptation](https://mlsanity.com/paper/2402.18334/) (2024)
- [AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering Benchmark Dataset](https://mlsanity.com/paper/2411.15640/) (2024)
- [AgroGPT: Efficient Agricultural Vision-Language Model with Expert Tuning](https://mlsanity.com/paper/2410.08405/) (2024)
- [WeatherQA: Can Multimodal Language Models Reason about Severe Weather?](https://mlsanity.com/paper/2406.11217/) (2024)
- [WildfireGPT: Tailored Large Language Model for Wildfire Analysis](https://mlsanity.com/paper/2402.07877/) (2024)
- [Accelerating Earth Science Discovery via Multi-Agent LLM Systems](https://mlsanity.com/paper/2503.05854/) (2025)
- [ClimateIQA: A New Dataset and Benchmark to Advance Vision-Language Models in Meteorology Anomalies Analysis](https://mlsanity.com/paper/2406.09838/) (2024)
- [AtmosSci-Bench: Evaluating the Recent Advance of Large Language Model for Atmospheric Science](https://mlsanity.com/paper/2502.01159/) (2025)
- [Exploring Large Language Models for Climate Forecasting](https://mlsanity.com/paper/2411.13724/) (2024)
- [Generative AI in Transportation Planning: A Survey](https://mlsanity.com/paper/2503.07158/) (2025)
- [Adapting While Learning: Grounding LLMs for Scientific Problems with Intelligent Tool Usage Adaptation](https://mlsanity.com/paper/2411.00412/) (2024)
- [Hyperbolic Large Language Models](https://mlsanity.com/paper/2509.05757/) (2025)
- [EAIRA: Establishing a Methodology for Evaluating AI Models as Scientific Research Assistants](https://mlsanity.com/paper/2502.20309/) (2025)
- [A RAG-Based Multi-Agent LLM System for Natural Hazard Resilience and Adaptation](https://mlsanity.com/paper/2504.17200/) (2025)
- [The Curious Language Model: Strategic Test-Time Information Acquisition](https://mlsanity.com/paper/2506.09173/) (2025)
- [Foundation Models for Environmental Science: A Survey of Emerging Frontiers](https://mlsanity.com/paper/2504.04280/) (2025)
- [Large Language Model Powered Intelligent Urban Agents: Concepts, Capabilities, and Applications](https://mlsanity.com/paper/2507.00914/) (2025)
- [MMESGBench: Pioneering Multimodal Understanding and Complex Reasoning Benchmark for ESG Tasks](https://mlsanity.com/paper/2507.18932/) (2025)
- [A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)](https://mlsanity.com/paper/2602.14696/) (2026)
- [Surveying Attitudinal Alignment Between Large Language Models Vs. Humans Towards 17 Sustainable Development Goals](https://mlsanity.com/paper/2404.13885/) (2024)
- [Assessing Generative Language Models in Classification Tasks: Performance and Self-Evaluation Capabilities in the Environmental and Climate Change Domain](https://mlsanity.com/paper/2408.17362/) (2024)
- [Prompting and Fine-Tuning of Small LLMs for Length-Controllable Telephone Call Summarization](https://mlsanity.com/paper/2410.18624/) (2024)
- [Towards deployment-centric multimodal AI beyond vision and language](https://mlsanity.com/paper/2504.03603/) (2025)
- [ClimateChat: Designing Data and Methods for Instruction Tuning LLMs to Answer Climate Change Queries](https://mlsanity.com/paper/2506.13796/) (2025)

## Abstract (arXiv)

This paper introduces ClimateGPT, a model family of domain-specific large language models that synthesize interdisciplinary research on climate change. We trained two 7B models from scratch on a science-oriented dataset of 300B tokens. For the first model, the 4.2B domain-specific tokens were included during pre-training and the second was adapted to the climate domain after pre-training. Additionally, ClimateGPT-7B, 13B and 70B are continuously pre-trained from Llama~2 on a domain-specific dataset of 4.2B tokens. Each model is instruction fine-tuned on a high-quality and human-generated domain-specific dataset that has been created in close cooperation with climate scientists. To reduce the number of hallucinations, we optimize the model for retrieval augmentation and propose a hierarchical retrieval strategy. To increase the accessibility of our model to non-English speakers, we propose to make use of cascaded machine translation and show that this approach can perform comparably to natively multilingual models while being easier to scale to a large number of languages. Further, to address the intrinsic interdisciplinary aspect of climate change we consider different research perspectives. Therefore, the model can produce in-depth answers focusing on different perspectives in addition to an overall answer. We propose a suite of automatic climate-specific benchmarks to evaluate LLMs. On these benchmarks, ClimateGPT-7B performs on par with the ten times larger Llama-2-70B Chat model while not degrading results on general domain benchmarks. Our human evaluation confirms the trends we saw in our benchmarks. All models were trained and evaluated using renewable energy and are released publicly.