# Emotion Ratings: How Intensity, Annotation Confidence and Agreements are Entangled

arXiv: https://arxiv.org/abs/2103.01667 · HTML: https://mlsanity.com/paper/2103.01667/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 33 references that are in the map.
- First posted on arXiv 2 Mar 2021.

## Builds on (most cited first, 7 of 7)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [Knowledge-Enriched Transformer for Emotion Detection in Textual Conversations](https://mlsanity.com/paper/1909.10681/) (2019)
- [Word Affect Intensities](https://mlsanity.com/paper/1704.08798/) (2017)
- [WASSA-2017 Shared Task on Emotion Intensity](https://mlsanity.com/paper/1708.03700/) (2017)
- [Appraisal Theories for Emotion Classification in Text](https://mlsanity.com/paper/2003.14155/) (2020)
- [Crowdsourcing and Validating Event-focused Emotion Corpora for German and English](https://mlsanity.com/paper/1905.13618/) (2019)

## Built on by (most cited first, 10 of 10)

- [Croissant: A Metadata Format for ML-Ready Datasets](https://mlsanity.com/paper/2403.19546/) (2024)
- [Empathic Conversations: A Multi-level Dataset of Contextualized Conversations](https://mlsanity.com/paper/2205.12698/) (2022)
- [Which Demographics do LLMs Default to During Annotation?](https://mlsanity.com/paper/2410.08820/) (2024)
- [Modeling Uncertainty in Personalized Emotion Prediction with Normalizing Flows](https://mlsanity.com/paper/2312.06034/) (2023)
- [Zero and Few-shot Learning for Author Profiling](https://mlsanity.com/paper/2204.10543/) (2022)
- [EmoTwiCS: A Corpus for Modelling Emotion Trajectories in Dutch Customer Service Dialogues on Twitter](https://mlsanity.com/paper/2310.06536/) (2023)
- [Artwork Interpretation with Vision Language Models: A Case Study on Emotions and Emotion Symbols](https://mlsanity.com/paper/2511.22929/) (2025)
- [Evaluating Theory of (an uncertain) Mind: Predicting the Uncertain Beliefs of Others in Conversation Forecasting](https://mlsanity.com/paper/2409.14986/) (2024)
- [Enhancing Multi-Label Emotion Analysis and Corresponding Intensities for Ethiopian Languages](https://mlsanity.com/paper/2503.18253/) (2025)
- [Daisy-TTS: Simulating Wider Spectrum of Emotions via Prosody Embedding Decomposition](https://mlsanity.com/paper/2402.14523/) (2024)

## Abstract (arXiv)

When humans judge the affective content of texts, they also implicitly assess the correctness of such judgment, that is, their confidence. We hypothesize that people's (in)confidence that they performed well in an annotation task leads to (dis)agreements among each other. If this is true, confidence may serve as a diagnostic tool for systematic differences in annotations. To probe our assumption, we conduct a study on a subset of the Corpus of Contemporary American English, in which we ask raters to distinguish neutral sentences from emotion-bearing ones, while scoring the confidence of their answers. Confidence turns out to approximate inter-annotator disagreements. Further, we find that confidence is correlated to emotion intensity: perceiving stronger affect in text prompts annotators to more certain classification performances. This insight is relevant for modelling studies of intensity, as it opens the question wether automatic regressors or classifiers actually predict intensity, or rather human's self-perceived confidence.