# Optimizing Large Language Models for ESG Activity Detection in Financial Texts

arXiv: https://arxiv.org/abs/2502.21112 · HTML: https://mlsanity.com/paper/2502.21112/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 55 references that are in the map.
- First posted on arXiv 28 Feb 2025.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [ORPO: Monolithic Preference Optimization without Reference Model](https://mlsanity.com/paper/2403.07691/) (2024)
- [FinGPT: Open-Source Financial Large Language Models](https://mlsanity.com/paper/2306.06031/) (2023)
- [Large Language Models in Finance: A Survey](https://mlsanity.com/paper/2311.10723/) (2023)
- [Knowledge Graphs: Opportunities and Challenges](https://mlsanity.com/paper/2303.13948/) (2023)
- [RecurrentGemma: Moving Past Transformers for Efficient Open Language Models](https://mlsanity.com/paper/2404.07839/) (2024)
- [MentaLLaMA: Interpretable Mental Health Analysis on Social Media with Large Language Models](https://mlsanity.com/paper/2309.13567/) (2023)
- [Artificial Intelligence for Literature Reviews: Opportunities and Challenges](https://mlsanity.com/paper/2402.08565/) (2024)
- [ESGReveal: An LLM-based approach for extracting structured data from ESG reports](https://mlsanity.com/paper/2312.17264/) (2023)
- [Weight Prediction Boosts the Convergence of AdamW](https://mlsanity.com/paper/2302.00195/) (2023)
- [Zero-Shot Question Answering over Financial Documents using Large Language Models](https://mlsanity.com/paper/2311.14722/) (2023)
- [Mining Patents with Large Language Models Elucidates the Chemical Function Landscape](https://mlsanity.com/paper/2309.08765/) (2023)

## Built on by (most cited first, 5 of 5)

- [SustainableQA: A Comprehensive Question Answering Dataset for Corporate Sustainability and EU Taxonomy Reporting](https://mlsanity.com/paper/2508.03000/) (2025)
- [Automated Analysis of Sustainability Reports: Using Large Language Models for the Extraction and Prediction of EU Taxonomy-Compliant KPIs](https://mlsanity.com/paper/2512.24289/) (2025)
- [A Comparative Study of Task Adaptation Techniques of Large Language Models for Identifying Sustainable Development Goals](https://mlsanity.com/paper/2506.15208/) (2025)
- [Integrating Large Language Models and Knowledge Graphs to Capture Political Viewpoints in News Media](https://mlsanity.com/paper/2512.14887/) (2025)
- [Empirical Evaluation of Open-Source Large Language Models for Retrieval-Augmented Generation in ESG Domain](https://mlsanity.com/paper/2609.15242/) (2026)

## Abstract (arXiv)

The integration of Environmental, Social, and Governance (ESG) factors into corporate decision-making is a fundamental aspect of sustainable finance. However, ensuring that business practices align with evolving regulatory frameworks remains a persistent challenge. AI-driven solutions for automatically assessing the alignment of sustainability reports and non-financial disclosures with specific ESG activities could greatly support this process. Yet, this task remains complex due to the limitations of general-purpose Large Language Models (LLMs) in domain-specific contexts and the scarcity of structured, high-quality datasets. In this paper, we investigate the ability of current-generation LLMs to identify text related to environmental activities. Furthermore, we demonstrate that their performance can be significantly enhanced through fine-tuning on a combination of original and synthetically generated data. To this end, we introduce ESG-Activities, a benchmark dataset containing 1,325 labelled text segments classified according to the EU ESG taxonomy. Our experimental results show that fine-tuning on ESG-Activities significantly enhances classification accuracy, with open models such as Llama 7B and Gemma 7B outperforming large proprietary solutions in specific configurations. These findings have important implications for financial analysts, policymakers, and AI researchers seeking to enhance ESG transparency and compliance through advanced natural language processing techniques.