# Transformer-based approaches to Sentiment Detection

arXiv: https://arxiv.org/abs/2303.07292 · HTML: https://mlsanity.com/paper/2303.07292/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 21 references that are in the map.
- First posted on arXiv 13 Mar 2023.

## Builds on (most cited first, 5 of 5)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [An Empirical Study on the Usage of BERT Models for Code Completion](https://mlsanity.com/paper/2103.07115/) (2021)

## Built on by (most cited first, 3 of 3)

- [Legend at ArAIEval Shared Task: Persuasion Technique Detection using a Language-Agnostic Text Representation Model](https://mlsanity.com/paper/2310.09661/) (2023)
- [MedAI Dialog Corpus (MEDIC): Zero-Shot Classification of Doctor and AI Responses in Health Consultations](https://mlsanity.com/paper/2310.12489/) (2023)
- [Evaluating Embeddings for One-Shot Classification of Doctor-AI Consultations](https://mlsanity.com/paper/2402.04442/) (2024)

## Abstract (arXiv)

The use of transfer learning methods is largely responsible for the present breakthrough in Natural Learning Processing (NLP) tasks across multiple domains. In order to solve the problem of sentiment detection, we examined the performance of four different types of well-known state-of-the-art transformer models for text classification. Models such as Bidirectional Encoder Representations from Transformers (BERT), Robustly Optimized BERT Pre-training Approach (RoBERTa), a distilled version of BERT (DistilBERT), and a large bidirectional neural network architecture (XLNet) were proposed. The performance of the four models that were used to detect disaster in the text was compared. All the models performed well enough, indicating that transformer-based models are suitable for the detection of disaster in text. The RoBERTa transformer model performs best on the test dataset with a score of 82.6% and is highly recommended for quality predictions. Furthermore, we discovered that the learning algorithms' performance was influenced by the pre-processing techniques, the nature of words in the vocabulary, unbalanced labeling, and the model parameters.