# ESGBERT: Language Model to Help with Classification Tasks Related to Companies Environmental, Social, and Governance Practices

arXiv: https://arxiv.org/abs/2203.16788 · HTML: https://mlsanity.com/paper/2203.16788/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 20 references that are in the map.
- First posted on arXiv 31 Mar 2022.

## Builds on (most cited first, 6 of 6)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018)
- [FinBERT: Financial Sentiment Analysis with Pre-trained Language Models](https://mlsanity.com/paper/1908.10063/) (2019)

## Built on by (most cited first, 11 of 11)

- [A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges](https://mlsanity.com/paper/2406.11903/) (2024)
- [Explainable Natural Language Processing for Corporate Sustainability Analysis](https://mlsanity.com/paper/2407.17487/) (2024)
- [Evaluating TCFD Reporting: A New Application of Zero-Shot Analysis to Climate-Related Financial Disclosures](https://mlsanity.com/paper/2302.00326/) (2023)
- [Leveraging BERT Language Models for Multi-Lingual ESG Issue Identification](https://mlsanity.com/paper/2309.02189/) (2023)
- [ESGLens: An LLM-Based RAG Framework for Interactive ESG Report Analysis and Score Prediction](https://mlsanity.com/paper/2604.19779/) (2026)
- [ESG-FTSE: A corpus of news articles with ESG relevance labels and use cases](https://mlsanity.com/paper/2405.20218/) (2024)
- [Environmental, Social and Governance Sentiment Analysis on Slovene News: A Novel Dataset and Models](https://mlsanity.com/paper/2604.06826/) (2026)
- [ESG Classification by Implicit Rule Learning via GPT-4](https://mlsanity.com/paper/2403.15040/) (2024)
- [Jetsons at FinNLP 2024: Towards Understanding the ESG Impact of a News Article using Transformer-based Models](https://mlsanity.com/paper/2404.00386/) (2024)
- [Polarity Detection of Sustainable Development Goals in News Text](https://mlsanity.com/paper/2509.19833/) (2025)
- [Domain-Adaptive Pretraining Enhances Water Treatment Semantic Representation for Large-Scale Structured Literature Mining](https://mlsanity.com/paper/2609.26034/) (2026)

## Abstract (arXiv)

Environmental, Social, and Governance (ESG) are non-financial factors that are garnering attention from investors as they increasingly look to apply these as part of their analysis to identify material risks and growth opportunities. Some of this attention is also driven by clients who, now more aware than ever, are demanding for their money to be managed and invested responsibly. As the interest in ESG grows, so does the need for investors to have access to consumable ESG information. Since most of it is in text form in reports, disclosures, press releases, and 10-Q filings, we see a need for sophisticated NLP techniques for classification tasks for ESG text. We hypothesize that an ESG domain-specific pre-trained model will help with such and study building of the same in this paper. We explored doing this by fine-tuning BERTs pre-trained weights using ESG specific text and then further fine-tuning the model for a classification task. We were able to achieve accuracy better than the original BERT and baseline models in environment-specific classification tasks.