# Fearful Falcons and Angry Llamas: Emotion Category Annotations of Arguments by Humans and LLMs

arXiv: https://arxiv.org/abs/2412.15993 · HTML: https://mlsanity.com/paper/2412.15993/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 54 references that are in the map.
- First posted on arXiv 20 Dec 2024.

## Builds on (most cited first, 21 of 21)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems](https://mlsanity.com/paper/1805.04508/) (2018)
- [How Many Data Points is a Prompt Worth?](https://mlsanity.com/paper/2103.08493/) (2021)
- [Argumentation Mining in User-Generated Web Discourse](https://mlsanity.com/paper/1601.02403/) (2016)
- [POTATO: The Portable Text Annotation Tool](https://mlsanity.com/paper/2212.08620/) (2022)
- [Dimensional Modeling of Emotions in Text with Appraisal Theories: Corpus Creation, Annotation Reliability, and Prediction](https://mlsanity.com/paper/2206.05238/) (2022)
- [Argument Strength is in the Eye of the Beholder: Audience Effects in Persuasion](https://mlsanity.com/paper/1708.09085/) (2017)
- [PRompt Optimization in Multi-Step Tasks (PROMST): Integrating Human Feedback and Heuristic-based Sampling](https://mlsanity.com/paper/2402.08702/) (2024)
- [Robust Prompt Optimization for Large Language Models Against Distribution Shifts](https://mlsanity.com/paper/2305.13954/) (2023)
- [From Once Upon a Time to Happily Ever After: Tracking Emotions in Novels and Fairy Tales](https://mlsanity.com/paper/1309.5909/) (2013)
- [Emotion Recognition under Consideration of the Emotion Component Process Model](https://mlsanity.com/paper/2107.12895/) (2021)
- [IEST: WASSA-2018 Implicit Emotions Shared Task](https://mlsanity.com/paper/1809.01083/) (2018)
- [Can Large Language Models Follow Concept Annotation Guidelines? A Case Study on Scientific and Financial Domains](https://mlsanity.com/paper/2311.08704/) (2023)
- [Can Factual Statements be Deceptive? The DeFaBel Corpus of Belief-based Deception](https://mlsanity.com/paper/2403.10185/) (2024)
- [TEII: Think, Explain, Interact and Iterate with Large Language Models to Solve Cross-lingual Emotion Detection](https://mlsanity.com/paper/2405.17129/) (2024)

## Built on by (most cited first, 11 of 11)

- [Do Emotions Really Affect Argument Convincingness? A Dynamic Approach with LLM-based Manipulation Checks](https://mlsanity.com/paper/2503.00024/) (2025)
- [LLMs Capture Emotion Labels, Not Emotion Uncertainty: Distributional Analysis and Calibration of Human-LLM Judgment Gaps](https://mlsanity.com/paper/2604.27345/) (2026)
- [Emotionally Charged, Logically Blurred: AI-driven Emotional Framing Impairs Human Fallacy Detection](https://mlsanity.com/paper/2510.09695/) (2025)
- [Investigating Subjective Factors of Argument Strength: Storytelling, Emotions, and Hedging](https://mlsanity.com/paper/2507.17409/) (2025)
- [Trust Me, I Can Convince You: The Contextualized Argument Appraisal Framework](https://mlsanity.com/paper/2509.17844/) (2025)
- [A Comparative Analysis of Word Segmentation, Part-of-Speech Tagging, and Named Entity Recognition for Historical Chinese Sources, 1900-1950](https://mlsanity.com/paper/2503.19844/) (2025)
- [Why Are Some Emotions Harder for LLMs? Uncovering the Causal Mechanisms of Emotion Inference via Sparse Autoencoders](https://mlsanity.com/paper/2604.25866/) (2026)
- [Categorical Emotions or Appraisals - Which Emotion Model Explains Argument Convincingness Better?](https://mlsanity.com/paper/2511.07162/) (2025)
- [Creation of the Estonian Subjectivity Dataset: Assessing the Degree of Subjectivity on a Scale](https://mlsanity.com/paper/2512.09634/) (2025)
- [From Emotion to Expression: Theoretical Foundations and Resources for Fear Speech](https://mlsanity.com/paper/2601.17132/) (2026)
- [Beyond the Rabbit Hole: Mapping the Relational Harms of QAnon Radicalization](https://mlsanity.com/paper/2601.17658/) (2026)

## Abstract (arXiv)

Arguments evoke emotions, influencing the effect of the argument itself. Not only the emotional intensity but also the category influence the argument's effects, for instance, the willingness to adapt stances. While binary emotionality has been studied in arguments, there is no work on discrete emotion categories (e.g., "Anger") in such data. To fill this gap, we crowdsource subjective annotations of emotion categories in a German argument corpus and evaluate automatic LLM-based labeling methods. Specifically, we compare three prompting strategies (zero-shot, one-shot, chain-of-thought) on three large instruction-tuned language models (Falcon-7b-instruct, Llama-3.1-8B-instruct, GPT-4o-mini). We further vary the definition of the output space to be binary (is there emotionality in the argument?), closed-domain (which emotion from a given label set is in the argument?), or open-domain (which emotion is in the argument?). We find that emotion categories enhance the prediction of emotionality in arguments, emphasizing the need for discrete emotion annotations in arguments. Across all prompt settings and models, automatic predictions show a high recall but low precision for predicting anger and fear, indicating a strong bias toward negative emotions.