# CliMedBERT: A Pre-trained Language Model for Climate and Health-related Text

arXiv: https://arxiv.org/abs/2212.00689 · HTML: https://mlsanity.com/paper/2212.00689/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 17 references that are in the map.
- First posted on arXiv 1 Dec 2022.

## Builds on (most cited first, 9 of 9)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [FinBERT: Financial Sentiment Analysis with Pre-trained Language Models](https://mlsanity.com/paper/1908.10063/) (2019)
- [CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims](https://mlsanity.com/paper/2012.00614/) (2020)
- [ClimateBert: A Pretrained Language Model for Climate-Related Text](https://mlsanity.com/paper/2110.12010/) (2021)
- [Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classification](https://mlsanity.com/paper/1908.11860/) (2019)
- [Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name Recognition](https://mlsanity.com/paper/2010.03746/) (2020)
- [Which early works are cited most frequently in climate change research literature? A bibliometric approach based on Reference Publication Year Spectroscopy](https://mlsanity.com/paper/1608.07960/) (2016)

## Built on by (most cited first, 5 of 5)

- [Foundation Models for Weather and Climate Data Understanding: A Comprehensive Survey](https://mlsanity.com/paper/2312.03014/) (2023)
- [ClimateNLP: Analyzing Public Sentiment Towards Climate Change Using Natural Language Processing](https://mlsanity.com/paper/2310.08099/) (2023)
- [Fine-tuning ClimateBert transformer with ClimaText for the disclosure analysis of climate-related financial risks](https://mlsanity.com/paper/2303.13373/) (2023)
- [On the Reliability of Large Language Models to Misinformed and Demographically-Informed Prompts](https://mlsanity.com/paper/2410.10850/) (2024)
- [Earth Science Foundation Models: From Perception to Reasoning and Discovery](https://mlsanity.com/paper/2605.12542/) (2026)

## Abstract (arXiv)

Climate change is threatening human health in unprecedented orders and many ways. These threats are expected to grow unless effective and evidence-based policies are developed and acted upon to minimize or eliminate them. Attaining such a task requires the highest degree of the flow of knowledge from science into policy. The multidisciplinary, location-specific, and vastness of published science makes it challenging to keep track of novel work in this area, as well as making the traditional knowledge synthesis methods inefficient in infusing science into policy. To this end, we consider developing multiple domain-specific language models (LMs) with different variations from Climate- and Health-related information, which can serve as a foundational step toward capturing available knowledge to enable solving different tasks, such as detecting similarities between climate- and health-related concepts, fact-checking, relation extraction, evidence of health effects to policy text generation, and more. To our knowledge, this is the first work that proposes developing multiple domain-specific language models for the considered domains. We will make the developed models, resources, and codebase available for the researchers.