# Rethinking Emotion Annotations in the Era of Large Language Models

arXiv: https://arxiv.org/abs/2412.07906 · HTML: https://mlsanity.com/paper/2412.07906/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning.
- Builds on 12 of its 76 references that are in the map.
- First posted on arXiv 10 Dec 2024.

## Placed in

- LLM reasoning / [Audio-language model reasoning](https://mlsanity.com/lr/audio-language-model-reasoning/)

## Builds on (most cited first, 12 of 12)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Large Language Models for Data Annotation and Synthesis: A Survey](https://mlsanity.com/paper/2402.13446/) (2024)
- [Quantifying the Persona Effect in LLM Simulations](https://mlsanity.com/paper/2402.10811/) (2024)
- [BERT: A Review of Applications in Natural Language Processing and Understanding](https://mlsanity.com/paper/2103.11943/) (2021)
- [Is ChatGPT Equipped with Emotional Dialogue Capabilities?](https://mlsanity.com/paper/2304.09582/) (2023)
- [Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBench](https://mlsanity.com/paper/2308.03656/) (2023)
- [TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models](https://mlsanity.com/paper/2311.17667/) (2023)
- [The Ambiguous World of Emotion Representation](https://mlsanity.com/paper/1909.00360/) (2019)
- [GPT-4 Emulates Average-Human Emotional Cognition from a Third-Person Perspective](https://mlsanity.com/paper/2408.13718/) (2024)
- [Can Large Language Models Aid in Annotating Speech Emotional Data? Uncovering New Frontiers](https://mlsanity.com/paper/2307.06090/) (2023)
- [Affective Computing for Healthcare: Recent Trends, Applications, Challenges, and Beyond](https://mlsanity.com/paper/2402.13589/) (2024)

## Built on by (most cited first, 8 of 8)

- [Large Language Models for Subjective Language Understanding: A Survey](https://mlsanity.com/paper/2508.07959/) (2025)
- [Scaling Ambiguity: Augmenting Human Annotation in Speech Emotion Recognition with Audio-Language Models](https://mlsanity.com/paper/2601.14620/) (2026)
- [The Limits of Learning from Pictures and Text: Vision-Language Models and Embodied Scene Understanding](https://mlsanity.com/paper/2603.26589/) (2026)
- [Switchboard-Affect: Emotion Perception Labels from Conversational Speech](https://mlsanity.com/paper/2510.13906/) (2025)
- [Affect Models Have Weak Generalizability to Atypical Speech](https://mlsanity.com/paper/2504.16283/) (2025)
- [ADEPT: RL-Aligned Agentic Decoding of Emotion via Evidence Probing Tools -- From Consensus Learning to Ambiguity-Driven Emotion Reasoning](https://mlsanity.com/paper/2602.12714/) (2026)
- [CEI: A Benchmark for Evaluating Pragmatic Reasoning in Language Models](https://mlsanity.com/paper/2603.09993/) (2026)
- [Whose Ground Truth? Embracing Ambiguity in Human-Centered AI](https://mlsanity.com/paper/2610.10805/) (2026)

## Abstract (arXiv)

Modern affective computing systems rely heavily on datasets with human-annotated emotion labels, for training and evaluation. However, human annotations are expensive to obtain, sensitive to study design, and difficult to quality control, because of the subjective nature of emotions. Meanwhile, Large Language Models (LLMs) have shown remarkable performance on many Natural Language Understanding tasks, emerging as a promising tool for text annotation. In this work, we analyze the complexities of emotion annotation in the context of LLMs, focusing on GPT-4 as a leading model. In our experiments, GPT-4 achieves high ratings in a human evaluation study, painting a more positive picture than previous work, in which human labels served as the only ground truth. On the other hand, we observe differences between human and GPT-4 emotion perception, underscoring the importance of human input in annotation studies. To harness GPT-4's strength while preserving human perspective, we explore two ways of integrating GPT-4 into emotion annotation pipelines, showing its potential to flag low-quality labels, reduce the workload of human annotators, and improve downstream model learning performance and efficiency. Together, our findings highlight opportunities for new emotion labeling practices and suggest the use of LLMs as a promising tool to aid human annotation.