mlsanityData to 14 Oct 2026

ClimateBert: A Pretrained Language Model for Climate-Related Text

arXiv 2110.12010 ↗First posted 22 Oct 2021cs.CLcited by 90

Not on the map. Why?

Abstract (arXiv)

Over the recent years, large pretrained language models (LM) have revolutionized the field of natural language processing (NLP). However, while pretraining on general language has been shown to work very well for common language, it has been observed that niche language poses problems. In particular, climate-related texts include specific language that common LMs can not represent accurately. We argue that this shortcoming of today's LMs limits the applicability of modern NLP to the broad field of text processing of climate-related texts. As a remedy, we propose CLIMATEBERT, a transformer-based language model that is further pretrained on over 2 million paragraphs of climate-related texts, crawled from various sources such as common news, research articles, and climate reporting of companies. We find that CLIMATEBERT leads to a 48% improvement on a masked language model objective which, in turn, leads to lowering error rates by 3.57% to 35.71% for various climate-related downstream tasks like text classification, sentiment analysis, and fact-checking.

Key facts. Cited by 90 papers in the map (data through 14 Oct 2026). Builds on 16 of its 32 references that are in the map. First posted on arXiv 22 Oct 2021.

Builds on

16 of 32 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  4. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  6. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  7. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  8. 2019FinBERT: Financial Sentiment Analysis with Pre-trained Language Modelscited by 385
  9. 2020Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease predictioncited by 373
  10. 2018A Simple Method for Commonsense Reasoningcited by 223
  11. 2020CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claimscited by 196
  12. 2017Learning to select data for transfer learning with Bayesian Optimizationcited by 64
4 more
  1. 2020ClimaText: A Dataset for Climate Change Topic Detectioncited by 22
  2. 2020Analyzing Sustainability Reports Using Natural Language Processingcited by 18
  3. 2015Climate change sentiment on Twitter: An unsolicited public opinion pollcited by 9
  4. 2021Automated Identification of Climate Risk Disclosures in Annual Corporate Reportscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 21; 2024: 24; 2025: 26; 2026: 15.

90 papers.

Sort:

By area:

  1. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  2. 2023Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directionscited by 55
  3. 2023Scientific Fact-Checking: A Survey of Resources and Approachescited by 54
  4. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 37
  5. 2023Foundation Models for Weather and Climate Data Understanding: A Comprehensive Surveycited by 34
  6. 2025Scientists' First Exam: Probing Cognitive Abilities of MLLM via Perception, Understanding, and Reasoningcited by 34
  7. 2024On the Foundations of Earth and Climate Foundation Modelscited by 33
  8. 2023chatClimate: Grounding Conversational AI in Climate Sciencecited by 32
  9. 2023CHATREPORT: Democratizing Sustainability Disclosure Analysis through LLM-based Toolscited by 29
  10. 2023Assessing Large Language Models on Climate Informationcited by 23
  11. 2023LLM for SoC Security: A Paradigm Shiftcited by 22
  12. 2024WeatherQA: Can Multimodal Language Models Reason about Severe Weather?cited by 22
38 more
  1. 2022Towards Climate Awareness in NLP Researchcited by 19
  2. 2024Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystackcited by 19
  3. 2023Enhancing Large Language Models with Climate Resourcescited by 18
  4. 2023Appraising the Potential Uses and Harms of LLMs for Medical Systematic Reviewscited by 18
  5. 2023ClimateBERT-NetZero: Detecting and Assessing Net Zero and Reduction Targetscited by 17
  6. 2024WeatherGFM: Learning A Weather Generalist Foundation Model via In-context Learningcited by 12
  7. 2022Environmental Claim Detectioncited by 11
  8. 2023Towards Answering Climate Questionnaires from Unstructured Climate Reportscited by 10
  9. 2024Laboratory-Scale AI: Open-Weight Models are Competitive with ChatGPT Even in Low-Resource Settingscited by 9
  10. 2025AtmosSci-Bench: Evaluating the Recent Advance of Large Language Model for Atmospheric Sciencecited by 9
  11. 2024Scalable Training of Trustworthy and Energy-Efficient Predictive Graph Foundation Models for Atomistic Materials Modeling: A Case Study with HydraGNNcited by 8
  12. 2024Exploring Large Language Models for Climate Forecastingcited by 8
  13. 2025ESGenius: Benchmarking LLMs on Environmental, Social, and Governance (ESG) and Sustainability Knowledgecited by 8
  14. 2024A Survey of Pre-trained Language Models for Processing Scientific Textcited by 7
  15. 2024Harnessing Business and Media Insights with Large Language Modelscited by 7
  16. 2025EarthSE: A Benchmark for Evaluating Earth Scientific Exploration Capability of LLMscited by 7
  17. 2023Glitter or Gold? Deriving Structured Insights from Sustainability Reports via Large Language Modelscited by 6
  18. 2023Climate Change from Large Language Modelscited by 6
  19. 2022CliMedBERT: A Pre-trained Language Model for Climate and Health-related Textcited by 5
  20. 2023ClimateNLP: Analyzing Public Sentiment Towards Climate Change Using Natural Language Processingcited by 5
  21. 2025CareerBERT: Matching Resumes to ESCO Jobs in a Shared Embedding Space for Generic Job Recommendationscited by 5
  22. 2025A RAG-Based Multi-Agent LLM System for Natural Hazard Resilience and Adaptationcited by 5
  23. 2025AutoClimDS: Climate Data Science Agentic AI -- A Knowledge Graph is All You Needcited by 5
  24. 2025Large Language Model Powered Intelligent Urban Agents: Concepts, Capabilities, and Applicationscited by 4
  25. 2023Evaluating TCFD Reporting: A New Application of Zero-Shot Analysis to Climate-Related Financial Disclosurescited by 3
  26. 2023Fine-tuning ClimateBert transformer with ClimaText for the disclosure analysis of climate-related financial riskscited by 3
  27. 2024EcoVerse: An Annotated Twitter Dataset for Eco-Relevance Classification, Environmental Impact Analysis, and Stance Detectioncited by 3
  28. 2024Surveying Attitudinal Alignment Between Large Language Models Vs. Humans Towards 17 Sustainable Development Goalscited by 3
  29. 2024Assessing Generative Language Models in Classification Tasks: Performance and Self-Evaluation Capabilities in the Environmental and Climate Change Domaincited by 3
  30. 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Modelscited by 3
  31. 2025ClimateChat: Designing Data and Methods for Instruction Tuning LLMs to Answer Climate Change Queriescited by 3
  32. 2025Multi-Agent Visual-Language Reasoning for Comprehensive Highway Scene Understandingcited by 3
  33. 2025Towards EnergyGPT: A Large Language Model Specialized for the Energy Sectorcited by 3
  34. 2025Survey of Multimodal Geospatial Foundation Models: Techniques, Applications, and Challengescited by 3
  35. 2023ESG Accountability Made Easy: DocQA at Your Servicecited by 2
  36. 2024Interpretable Cross-Examination Technique (ICE-T): Using highly informative features to boost LLM performancecited by 2
  37. 2024Multilingual Models for Check-Worthy Social Media Posts Detectioncited by 2
  38. 2024On the Reliability of Large Language Models to Misinformed and Demographically-Informed Promptscited by 2

50 most cited of 90 All 90 (JSON)