# From Text to Emotion: Unveiling the Emotion Annotation Capabilities of LLMs arXiv: https://arxiv.org/abs/2408.17026 ยท HTML: https://mlsanity.com/paper/2408.17026/ ## Key facts - Cited by 13 papers in the map (data through 14 Oct 2026). - Builds on 26 of its 41 references that are in the map. - First posted on arXiv 30 Aug 2024. ## Builds on (most cited first, 26 of 26) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022) - [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021) - [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021) - [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023) - [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022) - [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020) - [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022) - [SemEval-2017 Task 4: Sentiment Analysis in Twitter](https://mlsanity.com/paper/1912.00741/) (2019) - [Is GPT-3 a Good Data Annotator?](https://mlsanity.com/paper/2212.10450/) (2022) - [A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets](https://mlsanity.com/paper/2305.18486/) (2023) - [Text Classification via Large Language Models](https://mlsanity.com/paper/2305.08377/) (2023) - [EmoBank: Studying the Impact of Annotation Perspective and Representation Format on Dimensional Emotion Analysis](https://mlsanity.com/paper/2205.01996/) (2022) - [Emotional Intelligence of Large Language Models](https://mlsanity.com/paper/2307.09042/) (2023) - [Is ChatGPT Equipped with Emotional Dialogue Capabilities?](https://mlsanity.com/paper/2304.09582/) (2023) - [Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBench](https://mlsanity.com/paper/2308.03656/) (2023) - [EmoBench: Evaluating the Emotional Intelligence of Large Language Models](https://mlsanity.com/paper/2402.12071/) (2024) - [Comparing BERT against traditional machine learning text classification](https://mlsanity.com/paper/2005.13012/) (2020) - [Exploring the Sensitivity of LLMs' Decision-Making Capabilities: Insights from Prompt Variation and Hyperparameters](https://mlsanity.com/paper/2312.17476/) (2023) - [Foundation Model Assisted Automatic Speech Emotion Recognition: Transcribing, Annotating, and Augmenting](https://mlsanity.com/paper/2309.08108/) (2023) - [Refashioning Emotion Recognition Modelling: The Advent of Generalised Large Models](https://mlsanity.com/paper/2308.11578/) (2023) - [Estimating the Uncertainty in Emotion Attributes using Deep Evidential Regression](https://mlsanity.com/paper/2306.06760/) (2023) - [Bias in Emotion Recognition with ChatGPT](https://mlsanity.com/paper/2310.11753/) (2023) - [Can Large Language Models Aid in Annotating Speech Emotional Data? Uncovering New Frontiers](https://mlsanity.com/paper/2307.06090/) (2023) - [Affect Recognition in Conversations Using Large Language Models](https://mlsanity.com/paper/2309.12881/) (2023) ## Built on by (most cited first, 13 of 13) - [AER-LLM: Ambiguity-aware Emotion Recognition Leveraging Large Language Models](https://mlsanity.com/paper/2409.18339/) (2024) - [Richer Output for Richer Countries: Uncovering Geographical Disparities in Generated Stories and Travel Recommendations](https://mlsanity.com/paper/2411.07320/) (2024) - [Can Third-parties Read Our Emotions?](https://mlsanity.com/paper/2504.18673/) (2025) - [Leveraging Large Language Models for Sarcastic Speech Annotation in Sarcasm Detection](https://mlsanity.com/paper/2506.00955/) (2025) - [Decoding Ambiguous Emotions with Test-Time Scaling in Audio-Language Models](https://mlsanity.com/paper/2602.03873/) (2026) - [Labels Generated by Large Language Models Help Measure People's Empathy in Vitro](https://mlsanity.com/paper/2501.00691/) (2025) - [Switchboard-Affect: Emotion Perception Labels from Conversational Speech](https://mlsanity.com/paper/2510.13906/) (2025) - [Assessing the Reliability and Validity of GPT-4 in Annotating Emotion Appraisal Ratings](https://mlsanity.com/paper/2503.16883/) (2025) - [Affect Models Have Weak Generalizability to Atypical Speech](https://mlsanity.com/paper/2504.16283/) (2025) - [Contrastive Distillation of Emotion Knowledge from LLMs for Zero-Shot Emotion Recognition](https://mlsanity.com/paper/2505.18040/) (2025) - [SEER: The Span-based Emotion Evidence Retrieval Benchmark](https://mlsanity.com/paper/2510.03490/) (2025) - [Beyond the Rabbit Hole: Mapping the Relational Harms of QAnon Radicalization](https://mlsanity.com/paper/2601.17658/) (2026) - [ViGoEmotions: A Benchmark Dataset For Fine-grained Emotion Detection on Vietnamese Texts](https://mlsanity.com/paper/2602.08371/) (2026) ## Abstract (arXiv) Training emotion recognition models has relied heavily on human annotated data, which present diversity, quality, and cost challenges. In this paper, we explore the potential of Large Language Models (LLMs), specifically GPT4, in automating or assisting emotion annotation. We compare GPT4 with supervised models and or humans in three aspects: agreement with human annotations, alignment with human perception, and impact on model training. We find that common metrics that use aggregated human annotations as ground truth can underestimate the performance, of GPT-4 and our human evaluation experiment reveals a consistent preference for GPT-4 annotations over humans across multiple datasets and evaluators. Further, we investigate the impact of using GPT-4 as an annotation filtering process to improve model training. Together, our findings highlight the great potential of LLMs in emotion annotation tasks and underscore the need for refined evaluation methodologies.