# Enhanced Aspect-Based Sentiment Analysis Models with Progressive Self-supervised Attention Learning

arXiv: https://arxiv.org/abs/2103.03446 · HTML: https://mlsanity.com/paper/2103.03446/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 57 references that are in the map.
- First posted on arXiv 5 Mar 2021.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Is Attention Interpretable?](https://mlsanity.com/paper/1906.03731/) (2019)
- [DialogueRNN: An Attentive RNN for Emotion Detection in Conversations](https://mlsanity.com/paper/1811.00405/) (2018)
- [Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach](https://mlsanity.com/paper/1703.08448/) (2017)
- [Interactive Attention Networks for Aspect-Level Sentiment Classification](https://mlsanity.com/paper/1709.00893/) (2017)
- [Effective LSTMs for Target-Dependent Sentiment Classification](https://mlsanity.com/paper/1512.01100/) (2015)
- [Aspect Level Sentiment Classification with Deep Memory Network](https://mlsanity.com/paper/1605.08900/) (2016)
- [Open-Domain Targeted Sentiment Analysis via Span-Based Extraction and Classification](https://mlsanity.com/paper/1906.03820/) (2019)
- [Attentional Encoder Network for Targeted Sentiment Classification](https://mlsanity.com/paper/1902.09314/) (2019)
- [Transformation Networks for Target-Oriented Sentiment Classification](https://mlsanity.com/paper/1805.01086/) (2018)
- [Neural Machine Translation with Supervised Attention](https://mlsanity.com/paper/1609.04186/) (2016)
- [What does Attention in Neural Machine Translation Pay Attention to?](https://mlsanity.com/paper/1710.03348/) (2017)
- [Saliency-driven Word Alignment Interpretation for Neural Machine Translation](https://mlsanity.com/paper/1906.10282/) (2019)
- [Interpreting Recurrent and Attention-Based Neural Models: a Case Study on Natural Language Inference](https://mlsanity.com/paper/1808.03894/) (2018)
- [Sentiment and Sarcasm Classification with Multitask Learning](https://mlsanity.com/paper/1901.08014/) (2019)
- [Neural Simile Recognition with Cyclic Multitask Learning and Local Attention](https://mlsanity.com/paper/1912.09084/) (2019)
- [Progressive Self-Supervised Attention Learning for Aspect-Level Sentiment Analysis](https://mlsanity.com/paper/1906.01213/) (2019)
- [Who is Killed by Police: Introducing Supervised Attention for Hierarchical LSTMs](https://mlsanity.com/paper/1807.03409/) (2018)
- [From Random to Supervised: A Novel Dropout Mechanism Integrated with Global Information](https://mlsanity.com/paper/1808.08149/) (2018)

## Built on by (most cited first, 5 of 5)

- [A Survey on Aspect-Based Sentiment Classification](https://mlsanity.com/paper/2203.14266/) (2022)
- [A Review on Text-Based Emotion Detection -- Techniques, Applications, Datasets, and Future Directions](https://mlsanity.com/paper/2205.03235/) (2022)
- [Search-Engine-augmented Dialogue Response Generation with Cheaply Supervised Query Production](https://mlsanity.com/paper/2302.09300/) (2023)
- [Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Review](https://mlsanity.com/paper/2411.10268/) (2024)
- [Perturbation-based Self-supervised Attention for Attention Bias in Text Classification](https://mlsanity.com/paper/2305.15684/) (2023)

## Abstract (arXiv)

In aspect-based sentiment analysis (ABSA), many neural models are equipped with an attention mechanism to quantify the contribution of each context word to sentiment prediction. However, such a mechanism suffers from one drawback: only a few frequent words with sentiment polarities are tended to be taken into consideration for final sentiment decision while abundant infrequent sentiment words are ignored by models. To deal with this issue, we propose a progressive self-supervised attention learning approach for attentional ABSA models. In this approach, we iteratively perform sentiment prediction on all training instances, and continually learn useful attention supervision information in the meantime. During training, at each iteration, context words with the highest impact on sentiment prediction, identified based on their attention weights or gradients, are extracted as words with active/misleading influence on the correct/incorrect prediction for each instance. Words extracted in this way are masked for subsequent iterations. To exploit these extracted words for refining ABSA models, we augment the conventional training objective with a regularization term that encourages ABSA models to not only take full advantage of the extracted active context words but also decrease the weights of those misleading words. We integrate the proposed approach into three state-of-the-art neural ABSA models. Experiment results and in-depth analyses show that our approach yields better attention results and significantly enhances the performance of all three models. We release the source code and trained models at https://github.com/DeepLearnXMU/PSSAttention.