# Multilingual Twitter Sentiment Classification: The Role of Human Annotators

arXiv: https://arxiv.org/abs/1602.07563 · HTML: https://mlsanity.com/paper/1602.07563/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Feb 2016.

## Built on by (most cited first, 24 of 24)

- [NaijaSenti: A Nigerian Twitter Sentiment Corpus for Multilingual Sentiment Analysis](https://mlsanity.com/paper/2201.08277/) (2022)
- ["Garbage In, Garbage Out" Revisited: What Do Machine Learning Application Papers Report About Human-Labeled Training Data?](https://mlsanity.com/paper/2107.02278/) (2021)
- [SlovakBERT: Slovak Masked Language Model](https://mlsanity.com/paper/2109.15254/) (2021)
- [Massively Multilingual Corpus of Sentiment Datasets and Multi-faceted Sentiment Classification Benchmark](https://mlsanity.com/paper/2306.07902/) (2023)
- [Sequence to sequence pretraining for a less-resourced Slovenian language](https://mlsanity.com/paper/2207.13988/) (2022)
- [The ParlaSent Multilingual Training Dataset for Sentiment Identification in Parliamentary Proceedings](https://mlsanity.com/paper/2309.09783/) (2023)
- [Retweet communities reveal the main sources of hate speech](https://mlsanity.com/paper/2105.14898/) (2021)
- [A Study of Nationality Bias in Names and Perplexity using Off-the-Shelf Affect-related Tweet Classifiers](https://mlsanity.com/paper/2407.01834/) (2024)
- [Online Hate: Behavioural Dynamics and Relationship with Misinformation](https://mlsanity.com/paper/2105.14005/) (2021)
- [Cross-lingual alignments of ELMo contextual embeddings](https://mlsanity.com/paper/2106.15986/) (2021)
- [Assessment of Massively Multilingual Sentiment Classifiers](https://mlsanity.com/paper/2204.04937/) (2022)
- [The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages](https://mlsanity.com/paper/2310.14557/) (2023)
- [BERTuit: Understanding Spanish language in Twitter through a native transformer](https://mlsanity.com/paper/2204.03465/) (2022)
- [Russo-Ukrainian War: Prediction and explanation of Twitter suspension](https://mlsanity.com/paper/2306.03502/) (2023)
- [Creating emoji lexica from unsupervised sentiment analysis of their descriptions](https://mlsanity.com/paper/2404.01439/) (2024)
- [FinMarBa: A Market-Informed Dataset for Financial Sentiment Classification](https://mlsanity.com/paper/2507.22932/) (2025)
- [Current Landscape of the Russian Sentiment Corpora](https://mlsanity.com/paper/2106.14434/) (2021)
- [Cross-lingual Transfer for Text Classification with Dictionary-based Heterogeneous Graph](https://mlsanity.com/paper/2109.04400/) (2021)
- [PyTAIL: Interactive and Incremental Learning of NLP Models with Human in the Loop for Online Data](https://mlsanity.com/paper/2211.13786/) (2022)
- [Machine Translation for Accessible Multi-Language Text Analysis](https://mlsanity.com/paper/2301.08416/) (2023)
- [A Survey of Resources and Methods for Natural Language Processing of Serbian Language](https://mlsanity.com/paper/2304.05468/) (2023)
- [CroSentiNews 2.0: A Sentence-Level News Sentiment Corpus](https://mlsanity.com/paper/2305.08187/) (2023)
- [You Shall Know a Tool by the Traces it Leaves: The Predictability of Sentiment Analysis Tools](https://mlsanity.com/paper/2410.14626/) (2024)
- [What sentiment analysis can't see: Measuring whether customers were helped, and what went wrong, across 70,000 support conversations](https://mlsanity.com/paper/2606.19698/) (2026)

## Abstract (arXiv)

What are the limits of automated Twitter sentiment classification? We analyze a large set of manually labeled tweets in different languages, use them as training data, and construct automated classification models. It turns out that the quality of classification models depends much more on the quality and size of training data than on the type of the model trained. Experimental results indicate that there is no statistically significant difference between the performance of the top classification models. We quantify the quality of training data by applying various annotator agreement measures, and identify the weakest points of different datasets. We show that the model performance approaches the inter-annotator agreement when the size of the training set is sufficiently large. However, it is crucial to regularly monitor the self- and inter-annotator agreements since this improves the training datasets and consequently the model performance. Finally, we show that there is strong evidence that humans perceive the sentiment classes (negative, neutral, and positive) as ordered.