# Fear Is Not Always Fear: Variation in Emotion Label Equivalence Across Datasets

arXiv: https://arxiv.org/abs/2610.12841 · HTML: https://mlsanity.com/paper/2610.12841/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 34 references that are in the map.
- First posted on arXiv 9 Oct 2026.

## Builds on (most cited first, 16 of 16)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [MultiVerS: Improving scientific claim verification with weak supervision and full-document context](https://mlsanity.com/paper/2112.01640/) (2021)
- [On Releasing Annotator-Level Labels and Information in Datasets](https://mlsanity.com/paper/2110.05699/) (2021)
- [Measuring what Matters: Construct Validity in Large Language Model Benchmarks](https://mlsanity.com/paper/2511.04703/) (2025)
- [Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotation](https://mlsanity.com/paper/1712.01765/) (2017)
- [WASSA-2017 Shared Task on Emotion Intensity](https://mlsanity.com/paper/1708.03700/) (2017)
- [XED: A Multilingual Dataset for Sentiment Analysis and Emotion Detection](https://mlsanity.com/paper/2011.01612/) (2020)
- [Evaluating probabilistic classifiers: Reliability diagrams and score decompositions revisited](https://mlsanity.com/paper/2008.03033/) (2020)
- [Emotion Detection for Misinformation: A Review](https://mlsanity.com/paper/2311.00671/) (2023)
- [Experiencers, Stimuli, or Targets: Which Semantic Roles Enable Machine Learning to Infer the Emotions?](https://mlsanity.com/paper/2011.01599/) (2020)
- ["You are an expert annotator": Automatic Best-Worst-Scaling Annotations for Emotion Intensity Modeling](https://mlsanity.com/paper/2403.17612/) (2024)
- [Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework](https://mlsanity.com/paper/2606.21069/) (2026)

## Built on by (most cited first, 1 of 1)

- [Mixed Feelings, Single Polarity](https://mlsanity.com/paper/2610.15977/) (2026)

## Abstract (arXiv)

Emotion classifiers are routinely applied beyond their training data, on the assumption that same-named labels such as fear or surprise mean the same thing across resources. We test this assumption for five labels shared by GoEmotions, XED, and BRIGHTER, comparing models on identical inputs and under cross-corpus transfer. On the same texts, one fear classifier flags 71% as fearful and another 8%, yet the two largely agree on how the texts rank. Across 30 resource-label comparisons, training-label prevalence explains about half of this kind of scale mismatch, and a fitted intercept removes most of it. What recalibration cannot repair is larger: transferred models lose 55% of native discrimination in aggregate, agreement in ranking between classifiers is only weakly related to that loss, and for surprise the resources disagree about which texts express the emotion. The choice of training resource can therefore change conclusions: on two external tweet collections, BRIGHTER- and XED-trained classifiers give opposite answers about which shows more surprise, even after rank normalization, with intervals excluding zero in both directions. A shared emotion label name does not establish an interchangeable measurement target.