# RoBERTa-BiLSTM: A Context-Aware Hybrid Model for Sentiment Analysis

arXiv: https://arxiv.org/abs/2406.00367 · HTML: https://mlsanity.com/paper/2406.00367/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 101 references that are in the map.
- First posted on arXiv 1 Jun 2024.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [UL2: Unifying Language Learning Paradigms](https://mlsanity.com/paper/2205.05131/) (2022)
- [Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT](https://mlsanity.com/paper/2302.10198/) (2023)
- [On Empirical Comparisons of Optimizers for Deep Learning](https://mlsanity.com/paper/1910.05446/) (2019)
- [Unleashing the potential of prompt engineering for large language models](https://mlsanity.com/paper/2310.14735/) (2023)
- [Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research](https://mlsanity.com/paper/2005.00357/) (2020)
- [A Survey on Aspect-Based Sentiment Analysis: Tasks, Methods, and Challenges](https://mlsanity.com/paper/2203.01054/) (2022)
- [Deep Learning-based Sentiment Classification: A Comparative Survey](https://mlsanity.com/paper/2312.17253/) (2023)
- [Research on the Application of Deep Learning-based BERT Model in Sentiment Analysis](https://mlsanity.com/paper/2403.08217/) (2024)
- [Fast and Accurate FSA System Using ELBERT: An Efficient and Lightweight BERT](https://mlsanity.com/paper/2211.08842/) (2022)

## Built on by (most cited first, 7 of 7)

- [TWSSenti: A Novel Hybrid Framework for Topic-Wise Sentiment Analysis on Social Media Using Transformer Models](https://mlsanity.com/paper/2504.09896/) (2025)
- [Visualizing Public Opinion on X: A Real-Time Sentiment Dashboard Using VADER and DistilBERT](https://mlsanity.com/paper/2504.15448/) (2025)
- [DyFuLM: An Advanced Multimodal Framework for Sentiment Analysis](https://mlsanity.com/paper/2512.01410/) (2025)
- [Large Language Models for Education and Research: An Empirical and User Survey-based Analysis](https://mlsanity.com/paper/2512.08057/) (2025)
- [Entropy-Based Data Selection for Language Models](https://mlsanity.com/paper/2602.17465/) (2026)
- [AnySimLite: A Lightweight Few-Shot Similarity Encoder for On-Device Speech-Adjacent Classification](https://mlsanity.com/paper/2606.26452/) (2026)
- [Unveiling Public Opinion: A Study of Sentiment Analysis Using LSTM and Traditional Models](https://mlsanity.com/paper/2607.07772/) (2026)

## Abstract (arXiv)

Effectively analyzing the comments to uncover latent intentions holds immense value in making strategic decisions across various domains. However, several challenges hinder the process of sentiment analysis including the lexical diversity exhibited in comments, the presence of long dependencies within the text, encountering unknown symbols and words, and dealing with imbalanced datasets. Moreover, existing sentiment analysis tasks mostly leveraged sequential models to encode the long dependent texts and it requires longer execution time as it processes the text sequentially. In contrast, the Transformer requires less execution time due to its parallel processing nature. In this work, we introduce a novel hybrid deep learning model, RoBERTa-BiLSTM, which combines the Robustly Optimized BERT Pretraining Approach (RoBERTa) with Bidirectional Long Short-Term Memory (BiLSTM) networks. RoBERTa is utilized to generate meaningful word embedding vectors, while BiLSTM effectively captures the contextual semantics of long-dependent texts. The RoBERTa-BiLSTM hybrid model leverages the strengths of both sequential and Transformer models to enhance performance in sentiment analysis. We conducted experiments using datasets from IMDb, Twitter US Airline, and Sentiment140 to evaluate the proposed model against existing state-of-the-art methods. Our experimental findings demonstrate that the RoBERTa-BiLSTM model surpasses baseline models (e.g., BERT, RoBERTa-base, RoBERTa-GRU, and RoBERTa-LSTM), achieving accuracies of 80.74%, 92.36%, and 82.25% on the Twitter US Airline, IMDb, and Sentiment140 datasets, respectively. Additionally, the model achieves F1-scores of 80.73%, 92.35%, and 82.25% on the same datasets, respectively.