# An Empirical Study of the Effectiveness of an Ensemble of Stand-alone Sentiment Detection Tools for Software Engineering Datasets

arXiv: https://arxiv.org/abs/2111.03196 · HTML: https://mlsanity.com/paper/2111.03196/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 78 references that are in the map.
- First posted on arXiv 4 Nov 2021.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [A Benchmark Study on Sentiment Analysis for Software Engineering Research](https://mlsanity.com/paper/1803.06525/) (2018)
- [Sentiment Polarity Detection for Software Development](https://mlsanity.com/paper/1709.02984/) (2017)
- [EmoTxt: A Toolkit for Emotion Recognition from Text](https://mlsanity.com/paper/1708.03892/) (2017)
- [Can We Use SE-specific Sentiment Analysis Tools in a Cross-Platform Setting?](https://mlsanity.com/paper/2004.00300/) (2020)
- [Comparing and Combining Sentiment Analysis Methods](https://mlsanity.com/paper/1406.0032/) (2014)
- [Mining Valence, Arousal, and Dominance - Possibilities for Detecting Burnout and Productivity?](https://mlsanity.com/paper/1603.04287/) (2016)
- [SEntiMoji: An Emoji-Powered Learning Approach for Sentiment Analysis in Software Engineering](https://mlsanity.com/paper/1907.02202/) (2019)
- [A Gold Standard for Emotion Annotation in Stack Overflow](https://mlsanity.com/paper/1803.02300/) (2018)
- [Understanding How and Why Developers Seek and Analyze API-related Opinions](https://mlsanity.com/paper/2102.08495/) (2021)
- [Automatic API Usage Scenario Documentation from Technical Q&A Sites](https://mlsanity.com/paper/2102.08502/) (2021)

## Built on by (most cited first, 3 of 3)

- [An Empirical Study of IoT Security Aspects at Sentence-Level in Developer Textual Discussions](https://mlsanity.com/paper/2206.03079/) (2022)
- [LLM For Loop Invariant Generation and Fixing: How Far Are We?](https://mlsanity.com/paper/2511.06552/) (2025)
- [Sentiment analysis for software engineering: How far can zero-shot learning (ZSL) go?](https://mlsanity.com/paper/2604.13826/) (2026)

## Abstract (arXiv)

Sentiment analysis in software engineering (SE) has shown promise to analyze and support diverse development activities. We report the results of an empirical study that we conducted to determine the feasibility of developing an ensemble engine by combining the polarity labels of stand-alone SE-specific sentiment detectors. Our study has two phases. In the first phase, we pick five SE-specific sentiment detection tools from two recently published papers by Lin et al. [31, 32], who first reported negative results with standalone sentiment detectors and then proposed an improved SE-specific sentiment detector, POME [31]. We report the study results on 17,581 units (sentences/documents) coming from six currently available sentiment benchmarks for SE. We find that the existing tools can be complementary to each other in 85-95% of the cases, i.e., one is wrong, but another is right. However, a majority voting-based ensemble of those tools fails to improve the accuracy of sentiment detection. We develop Sentisead, a supervised tool by combining the polarity labels and bag of words as features. Sentisead improves the performance (F1-score) of the individual tools by 4% (over Senti4SD [5]) - 100% (over POME [31]). In a second phase, we compare and improve Sentisead infrastructure using Pre-trained Transformer Models (PTMs). We find that a Sentisead infrastructure with RoBERTa as the ensemble of the five stand-alone rule-based and shallow learning SE-specific tools from Lin et al. [31, 32] offers the best F1-score of 0.805 across the six datasets, while a stand-alone RoBERTa shows an F1-score of 0.801.