# Sentiment analysis is not solved! Assessing and probing sentiment classification

arXiv: https://arxiv.org/abs/1906.05887 · HTML: https://mlsanity.com/paper/1906.05887/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jun 2019.

## Built on by (most cited first, 14 of 14)

- [XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond](https://mlsanity.com/paper/2104.12250/) (2021)
- [Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future](https://mlsanity.com/paper/2206.02280/) (2022)
- [Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models](https://mlsanity.com/paper/2103.08849/) (2021)
- [CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation](https://mlsanity.com/paper/2211.00295/) (2022)
- [The King is Naked: on the Notion of Robustness for Natural Language Processing](https://mlsanity.com/paper/2112.07605/) (2021)
- [Decoding EEG Brain Activity for Multi-Modal Natural Language Processing](https://mlsanity.com/paper/2102.08655/) (2021)
- [Language Models as a Service: Overview of a New Paradigm and its Challenges](https://mlsanity.com/paper/2309.16573/) (2023)
- [On Efficient and Statistical Quality Estimation for Data Annotation](https://mlsanity.com/paper/2405.11919/) (2024)
- [Capabilities for Better ML Engineering](https://mlsanity.com/paper/2211.06409/) (2022)
- [Recursive Neural Networks with Bottlenecks Diagnose (Non-)Compositionality](https://mlsanity.com/paper/2301.13714/) (2023)
- [Non-Compositionality in Sentiment: New Data and Analyses](https://mlsanity.com/paper/2310.20656/) (2023)
- [Does BERT look at sentiment lexicon?](https://mlsanity.com/paper/2111.10100/) (2021)
- [Generalizable Error Modeling for Human Data Annotation: Evidence From an Industry-Scale Search Data Annotation Program](https://mlsanity.com/paper/2310.05286/) (2023)
- [Noise in Relation Classification Dataset TACRED: Characterization and Reduction](https://mlsanity.com/paper/2311.12298/) (2023)

## Abstract (arXiv)

Neural methods for SA have led to quantitative improvements over previous approaches, but these advances are not always accompanied with a thorough analysis of the qualitative differences. Therefore, it is not clear what outstanding conceptual challenges for sentiment analysis remain. In this work, we attempt to discover what challenges still prove a problem for sentiment classifiers for English and to provide a challenging dataset. We collect the subset of sentences that an (oracle) ensemble of state-of-the-art sentiment classifiers misclassify and then annotate them for 18 linguistic and paralinguistic phenomena, such as negation, sarcasm, modality, etc. The dataset is available at https://github.com/ltgoslo/assessing_and_probing_sentiment. Finally, we provide a case study that demonstrates the usefulness of the dataset to probe the performance of a given sentiment classifier with respect to linguistic phenomena.