# BERT-Based Sentiment Analysis: A Software Engineering Perspective

arXiv: https://arxiv.org/abs/2106.02581 · HTML: https://mlsanity.com/paper/2106.02581/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 34 references that are in the map.
- First posted on arXiv 4 Jun 2021.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method](https://mlsanity.com/paper/1402.3722/) (2014)
- [Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features](https://mlsanity.com/paper/1703.02507/) (2017)
- [Natural Language Processing Advancements By Deep Learning: A Survey](https://mlsanity.com/paper/2003.01200/) (2020)
- [A Benchmark Study on Sentiment Analysis for Software Engineering Research](https://mlsanity.com/paper/1803.06525/) (2018)
- [Sentiment Polarity Detection for Software Development](https://mlsanity.com/paper/1709.02984/) (2017)
- [Data Augmentation in a Hybrid Approach for Aspect-Based Sentiment Analysis](https://mlsanity.com/paper/2103.15912/) (2021)
- [EmoTxt: A Toolkit for Emotion Recognition from Text](https://mlsanity.com/paper/1708.03892/) (2017)
- [SEntiMoji: An Emoji-Powered Learning Approach for Sentiment Analysis in Software Engineering](https://mlsanity.com/paper/1907.02202/) (2019)
- [Recommending Insightful Comments for Source Code using Crowdsourced Knowledge](https://mlsanity.com/paper/1807.02278/) (2018)
- [Pattern Learning for Detecting Defect Reports and Improvement Requests in App Reviews](https://mlsanity.com/paper/2004.08793/) (2020)

## Built on by (most cited first, 9 of 9)

- [Securing Large Language Models: Threats, Vulnerabilities and Responsible Practices](https://mlsanity.com/paper/2403.12503/) (2024)
- [Decoding the AI Pen: Techniques and Challenges in Detecting AI-Generated Text](https://mlsanity.com/paper/2403.05750/) (2024)
- [Hate speech detection using static BERT embeddings](https://mlsanity.com/paper/2106.15537/) (2021)
- [TransformCode: A Contrastive Learning Framework for Code Embedding via Subtree Transformation](https://mlsanity.com/paper/2311.08157/) (2023)
- [An Empirical Study of IoT Security Aspects at Sentence-Level in Developer Textual Discussions](https://mlsanity.com/paper/2206.03079/) (2022)
- [Concept-Based Interpretability for Toxicity Detection](https://mlsanity.com/paper/2511.16689/) (2025)
- [Tricks and Plug-ins for Gradient Boosting with Transformers](https://mlsanity.com/paper/2508.02924/) (2025)
- [Polarity Detection of Sustainable Development Goals in News Text](https://mlsanity.com/paper/2509.19833/) (2025)
- [Sentiment analysis for software engineering: How far can zero-shot learning (ZSL) go?](https://mlsanity.com/paper/2604.13826/) (2026)

## Abstract (arXiv)

Sentiment analysis can provide a suitable lead for the tools used in software engineering along with the API recommendation systems and relevant libraries to be used. In this context, the existing tools like SentiCR, SentiStrength-SE, etc. exhibited low f1-scores that completely defeats the purpose of deployment of such strategies, thereby there is enough scope for performance improvement. Recent advancements show that transformer based pre-trained models (e.g., BERT, RoBERTa, ALBERT, etc.) have displayed better results in the text classification task. Following this context, the present research explores different BERT-based models to analyze the sentences in GitHub comments, Jira comments, and Stack Overflow posts. The paper presents three different strategies to analyse BERT based model for sentiment analysis, where in the first strategy the BERT based pre-trained models are fine-tuned; in the second strategy an ensemble model is developed from BERT variants, and in the third strategy a compressed model (Distil BERT) is used. The experimental results show that the BERT based ensemble approach and the compressed BERT model attain improvements by 6-12% over prevailing tools for the F1 measure on all three datasets.