# Does BERT look at sentiment lexicon?

arXiv: https://arxiv.org/abs/2111.10100 · HTML: https://mlsanity.com/paper/2111.10100/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 38 references that are in the map.
- First posted on arXiv 19 Nov 2021.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Crowdsourcing a Word-Emotion Association Lexicon](https://mlsanity.com/paper/1308.6297/) (2013)
- [Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language](https://mlsanity.com/paper/1905.07213/) (2019)
- [How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking](https://mlsanity.com/paper/2004.14992/) (2020)
- [Interpretation of NLP models through input marginalization](https://mlsanity.com/paper/2010.13984/) (2020)
- [Morphological Analyzer and Generator for Russian and Ukrainian Languages](https://mlsanity.com/paper/1503.07283/) (2015)
- [On Explaining Your Explanations of BERT: An Empirical Study with Sequence Classification](https://mlsanity.com/paper/2101.00196/) (2021)
- [Modeling emotion in complex stories: the Stanford Emotional Narratives Dataset](https://mlsanity.com/paper/1912.05008/) (2019)
- [Sentiment analysis is not solved! Assessing and probing sentiment classification](https://mlsanity.com/paper/1906.05887/) (2019)
- [Structured Self-Attention Weights Encode Semantics in Sentiment Analysis](https://mlsanity.com/paper/2010.04922/) (2020)

## Built on by (most cited first, 1 of 1)

- [Deep Knowledge-Infusion For Explainable Depression Detection](https://mlsanity.com/paper/2409.02122/) (2024)

## Abstract (arXiv)

The main approaches to sentiment analysis are rule-based methods and ma-chine learning, in particular, deep neural network models with the Trans-former architecture, including BERT. The performance of neural network models in the tasks of sentiment analysis is superior to the performance of rule-based methods. The reasons for this situation remain unclear due to the poor interpretability of deep neural network models. One of the main keys to understanding the fundamental differences between the two approaches is the analysis of how sentiment lexicon is taken into account in neural network models. To this end, we study the attention weights matrices of the Russian-language RuBERT model. We fine-tune RuBERT on sentiment text corpora and compare the distributions of attention weights for sentiment and neutral lexicons. It turns out that, on average, 3/4 of the heads of various model var-iants statistically pay more attention to the sentiment lexicon compared to the neutral one.