# Sentiment analysis with adaptive multi-head attention in Transformer

arXiv: https://arxiv.org/abs/2310.14505 · HTML: https://mlsanity.com/paper/2310.14505/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 1 of its 6 references that are in the map.
- First posted on arXiv 23 Oct 2023.

## Builds on (most cited first, 1 of 1)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)

## Built on by (most cited first, 5 of 5)

- [Evolution and Efficiency in Neural Architecture Search: Bridging the Gap Between Expert Design and Automated Optimization](https://mlsanity.com/paper/2403.17012/) (2024)
- [FedEmb: A Vertical and Hybrid Federated Learning Algorithm using Network And Feature Embedding Aggregation](https://mlsanity.com/paper/2312.00102/) (2023)
- [Joint Detection Algorithm for Multiple Cognitive Users in Spectrum Sensing](https://mlsanity.com/paper/2311.18599/) (2023)
- [Optimizing the Passenger Flow for Airport Security Check](https://mlsanity.com/paper/2312.05259/) (2023)
- [Dynamic Adaptive Attention and Supervised Contrastive Learning: A Novel Hybrid Framework for Text Sentiment Classification](https://mlsanity.com/paper/2604.10459/) (2026)

## Abstract (arXiv)

We propose a novel framework based on the attention mechanism to identify the sentiment of a movie review document. Previous efforts on deep neural networks with attention mechanisms focus on encoder and decoder with fixed numbers of multi-head attention. Therefore, we need a mechanism to stop the attention process automatically if no more useful information can be read from the memory.In this paper, we propose an adaptive multi-head attention architecture (AdaptAttn) which varies the number of attention heads based on length of sentences. AdaptAttn has a data preprocessing step where each document is classified into any one of the three bins small, medium or large based on length of the sentence. The document classified as small goes through two heads in each layer, the medium group passes four heads and the large group is processed by eight heads. We examine the merit of our model on the Stanford large movie review dataset. The experimental results show that the F1 score from our model is on par with the baseline model.