# Emotion-Aware, Emotion-Agnostic, or Automatic: Corpus Creation Strategies to Obtain Cognitive Event Appraisal Annotations

arXiv: https://arxiv.org/abs/2102.12858 · HTML: https://mlsanity.com/paper/2102.12858/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 44 references that are in the map.
- First posted on arXiv 25 Feb 2021.

## Builds on (most cited first, 10 of 10)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)
- [NRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets](https://mlsanity.com/paper/1308.6242/) (2013)
- [Appraisal Theories for Emotion Classification in Text](https://mlsanity.com/paper/2003.14155/) (2020)
- [Crowdsourcing and Validating Event-focused Emotion Corpora for German and English](https://mlsanity.com/paper/1905.13618/) (2019)
- [ktrain: A Low-Code Library for Augmented Machine Learning](https://mlsanity.com/paper/2004.10703/) (2020)
- [PO-EMO: Conceptualization, Annotation, and Modeling of Aesthetic Emotions in German and English Poetry](https://mlsanity.com/paper/2003.07723/) (2020)
- [A Survey on Sentiment and Emotion Analysis for Computational Literary Studies](https://mlsanity.com/paper/1808.03137/) (2018)
- [Towards Multimodal Emotion Recognition in German Speech Events in Cars using Transfer Learning](https://mlsanity.com/paper/1909.02764/) (2019)

## Built on by (most cited first, 6 of 6)

- [Dimensional Modeling of Emotions in Text with Appraisal Theories: Corpus Creation, Annotation Reliability, and Prediction](https://mlsanity.com/paper/2206.05238/) (2022)
- [x-enVENT: A Corpus of Event Descriptions with Experiencer-specific Emotion and Appraisal Annotations](https://mlsanity.com/paper/2203.10909/) (2022)
- [APPReddit: a Corpus of Reddit Posts Annotated for Appraisal](https://mlsanity.com/paper/2205.15627/) (2022)
- [Where are We in Event-centric Emotion Analysis? Bridging Emotion Role Labeling and Appraisal-based Approaches](https://mlsanity.com/paper/2309.02092/) (2023)
- [Emotion Stimulus Detection in German News Headlines](https://mlsanity.com/paper/2107.12920/) (2021)
- [Assessing the Reliability and Validity of GPT-4 in Annotating Emotion Appraisal Ratings](https://mlsanity.com/paper/2503.16883/) (2025)

## Abstract (arXiv)

Appraisal theories explain how the cognitive evaluation of an event leads to a particular emotion. In contrast to theories of basic emotions or affect (valence/arousal), this theory has not received a lot of attention in natural language processing. Yet, in psychology it has been proven powerful: Smith and Ellsworth (1985) showed that the appraisal dimensions attention, certainty, anticipated effort, pleasantness, responsibility/control and situational control discriminate between (at least) 15 emotion classes. We study different annotation strategies for these dimensions, based on the event-focused enISEAR corpus (Troiano et al., 2019). We analyze two manual annotation settings: (1) showing the text to annotate while masking the experienced emotion label; (2) revealing the emotion associated with the text. Setting 2 enables the annotators to develop a more realistic intuition of the described event, while Setting 1 is a more standard annotation procedure, purely relying on text. We evaluate these strategies in two ways: by measuring inter-annotator agreement and by fine-tuning RoBERTa to predict appraisal variables. Our results show that knowledge of the emotion increases annotators' reliability. Further, we evaluate a purely automatic rule-based labeling strategy (inferring appraisal from annotated emotion classes). Training on automatically assigned labels leads to a competitive performance of our classifier, even when tested on manual annotations. This is an indicator that it might be possible to automatically create appraisal corpora for every domain for which emotion corpora already exist.