# Assessing Generative Language Models in Classification Tasks: Performance and Self-Evaluation Capabilities in the Environmental and Climate Change Domain

arXiv: https://arxiv.org/abs/2408.17362 · HTML: https://mlsanity.com/paper/2408.17362/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 30 Aug 2024.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [Sentiment Analysis in the Era of Large Language Models: A Reality Check](https://mlsanity.com/paper/2305.15005/) (2023)
- [ClimateBert: A Pretrained Language Model for Climate-Related Text](https://mlsanity.com/paper/2110.12010/) (2021)
- [Self-Evaluation Improves Selective Generation in Large Language Models](https://mlsanity.com/paper/2312.09300/) (2023)
- [ClimateGPT: Towards AI Synthesizing Interdisciplinary Research on Climate Change](https://mlsanity.com/paper/2401.09646/) (2024)
- [Assessing Large Language Models on Climate Information](https://mlsanity.com/paper/2310.02932/) (2023)
- [Towards Climate Awareness in NLP Research](https://mlsanity.com/paper/2205.05071/) (2022)
- [Evaluation of ChatGPT Family of Models for Biomedical Reasoning and Classification](https://mlsanity.com/paper/2304.02496/) (2023)
- [Fairness of ChatGPT and the Role Of Explainable-Guided Prompts](https://mlsanity.com/paper/2307.11761/) (2023)
- [A Multi-task Model for Sentiment Aided Stance Detection of Climate Change Tweets](https://mlsanity.com/paper/2211.03533/) (2022)
- [Large Language Models for Propaganda Detection](https://mlsanity.com/paper/2310.06422/) (2023)
- [Climate Change from Large Language Models](https://mlsanity.com/paper/2312.11985/) (2023)
- [Automated Fact-Checking of Climate Change Claims with Large Language Models](https://mlsanity.com/paper/2401.12566/) (2024)
- [EcoVerse: An Annotated Twitter Dataset for Eco-Relevance Classification, Environmental Impact Analysis, and Stance Detection](https://mlsanity.com/paper/2404.05133/) (2024)

## Built on by (most cited first, 3 of 3)

- [ScholaWrite: A Dataset of End-to-End Scholarly Writing Process](https://mlsanity.com/paper/2502.02904/) (2025)
- [LLaVul: A Multimodal LLM for Interpretable Vulnerability Reasoning about Source Code](https://mlsanity.com/paper/2509.17337/) (2025)
- [FITS: Towards an AI-Driven Fashion Information Tool for Sustainability](https://mlsanity.com/paper/2509.26017/) (2025)

## Abstract (arXiv)

This paper examines the performance of two Large Language Models (LLMs), GPT3.5 and Llama2 and one Small Language Model (SLM) Gemma, across three different classification tasks within the climate change (CC) and environmental domain. Employing BERT-based models as a baseline, we compare their efficacy against these transformer-based models. Additionally, we assess the models' self-evaluation capabilities by analyzing the calibration of verbalized confidence scores in these text classification tasks. Our findings reveal that while BERT-based models generally outperform both the LLMs and SLM, the performance of the large generative models is still noteworthy. Furthermore, our calibration analysis reveals that although Gemma is well-calibrated in initial tasks, it thereafter produces inconsistent results; Llama is reasonably calibrated, and GPT consistently exhibits strong calibration. Through this research, we aim to contribute to the ongoing discussion on the utility and effectiveness of generative LMs in addressing some of the planet's most urgent issues, highlighting their strengths and limitations in the context of ecology and CC.