# Sentiment Analysis in Software Engineering: Evaluating Generative Pre-trained Transformers

arXiv: https://arxiv.org/abs/2505.14692 · HTML: https://mlsanity.com/paper/2505.14692/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 24 references that are in the map.
- First posted on arXiv 22 Apr 2025.

## Builds on (most cited first, 11 of 11)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [A Benchmark Study on Sentiment Analysis for Software Engineering Research](https://mlsanity.com/paper/1803.06525/) (2018)
- [Sentiment Polarity Detection for Software Development](https://mlsanity.com/paper/1709.02984/) (2017)
- [EmoTxt: A Toolkit for Emotion Recognition from Text](https://mlsanity.com/paper/1708.03892/) (2017)
- [Can We Use SE-specific Sentiment Analysis Tools in a Cross-Platform Setting?](https://mlsanity.com/paper/2004.00300/) (2020)
- [A Gold Standard for Emotion Annotation in Stack Overflow](https://mlsanity.com/paper/1803.02300/) (2018)
- [Expressions of Sentiments During Code Reviews: Male vs. Female](https://mlsanity.com/paper/1812.05560/) (2018)
- [Sentiment Analysis Tools in Software Engineering: A Systematic Mapping Study](https://mlsanity.com/paper/2502.07893/) (2025)

## Abstract (arXiv)

Sentiment analysis plays a crucial role in understanding developer interactions, issue resolutions, and project dynamics within software engineering (SE). While traditional SE-specific sentiment analysis tools have made significant strides, they often fail to account for the nuanced and context-dependent language inherent to the domain. This study systematically evaluates the performance of bidirectional transformers, such as BERT, against generative pre-trained transformers, specifically GPT-4o-mini, in SE sentiment analysis. Using datasets from GitHub, Stack Overflow, and Jira, we benchmark the models' capabilities with fine-tuned and default configurations. The results reveal that fine-tuned GPT-4o-mini performs comparable to BERT and other bidirectional models on structured and balanced datasets like GitHub and Jira, achieving macro-averaged F1-scores of 0.93 and 0.98, respectively. However, on linguistically complex datasets with imbalanced sentiment distributions, such as Stack Overflow, the default GPT-4o-mini model exhibits superior generalization, achieving an accuracy of 85.3% compared to the fine-tuned model's 13.1%. These findings highlight the trade-offs between fine-tuning and leveraging pre-trained models for SE tasks. The study underscores the importance of aligning model architectures with dataset characteristics to optimize performance and proposes directions for future research in refining sentiment analysis tools tailored to the SE domain.