# Can Third-parties Read Our Emotions?

arXiv: https://arxiv.org/abs/2504.18673 · HTML: https://mlsanity.com/paper/2504.18673/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 65 references that are in the map.
- First posted on arXiv 25 Apr 2025.

## Builds on (most cited first, 23 of 23)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)
- [Aligning Language Models to User Opinions](https://mlsanity.com/paper/2305.14929/) (2023)
- [Stance Detection on Social Media: State of the Art and Trends](https://mlsanity.com/paper/2006.03644/) (2020)
- [CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation](https://mlsanity.com/paper/2310.15638/) (2023)
- [Toxicity Detection: Does Context Really Matter?](https://mlsanity.com/paper/2006.00998/) (2020)
- [Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting](https://mlsanity.com/paper/2309.07034/) (2023)
- [MEGAnno+: A Human-LLM Collaborative Annotation System](https://mlsanity.com/paper/2402.18050/) (2024)
- [iSarcasm: A Dataset of Intended Sarcasm](https://mlsanity.com/paper/1911.03123/) (2019)
- [Angry Men, Sad Women: Large Language Models Reflect Gendered Stereotypes in Emotion Attribution](https://mlsanity.com/paper/2403.03121/) (2024)
- [Emotion Intensities in Tweets](https://mlsanity.com/paper/1708.03696/) (2017)
- [Cultural Conditioning or Placebo? On the Effectiveness of Socio-Demographic Prompting](https://mlsanity.com/paper/2406.11661/) (2024)
- [Large Language Models for Propaganda Span Annotation](https://mlsanity.com/paper/2311.09812/) (2023)
- [DENS: A Dataset for Multi-class Emotion Analysis](https://mlsanity.com/paper/1910.11769/) (2019)
- [Emotion fusion for mental illness detection from social media: A survey](https://mlsanity.com/paper/2304.09493/) (2023)
- [From Text to Emotion: Unveiling the Emotion Annotation Capabilities of LLMs](https://mlsanity.com/paper/2408.17026/) (2024)
- [The Sentiment Problem: A Critical Survey towards Deconstructing Sentiment Analysis](https://mlsanity.com/paper/2310.12318/) (2023)
- [(Mis)alignment Between Stance Expressed in Social Media Data and Public Opinion Surveys](https://mlsanity.com/paper/2109.01762/) (2021)
- [Advancing Annotation of Stance in Social Media Posts: A Comparative Analysis of Large Language Models and Crowd Sourcing](https://mlsanity.com/paper/2406.07483/) (2024)

## Built on by (most cited first, 4 of 4)

- [Affective Context Amplifies Sycophancy in LLM Responses](https://mlsanity.com/paper/2608.21242/) (2026)
- [What You Feel Is Not What They See: On Predicting Self-Reported Emotion from Third-Party Observer Labels](https://mlsanity.com/paper/2601.21130/) (2026)
- [A Customer Journey in the Land of Oz: Leveraging the Wizard of Oz Technique to Model Emotions in Customer Service Interactions](https://mlsanity.com/paper/2511.21909/) (2025)
- [Emotion Experience, Expression, and Perception: Emotion Analysis on Multimodal Social Media Posts](https://mlsanity.com/paper/2609.18385/) (2026)

## Abstract (arXiv)

Natural Language Processing tasks that aim to infer an author's private states, e.g., emotions and opinions, from their written text, typically rely on datasets annotated by third-party annotators. However, the assumption that third-party annotators can accurately capture authors' private states remains largely unexamined. In this study, we present human subjects experiments on emotion recognition tasks that directly compare third-party annotations with first-party (author-provided) emotion labels. Our findings reveal significant limitations in third-party annotations-whether provided by human annotators or large language models (LLMs)-in faithfully representing authors' private states. However, LLMs outperform human annotators nearly across the board. We further explore methods to improve third-party annotation quality. We find that demographic similarity between first-party authors and third-party human annotators enhances annotation performance. While incorporating first-party demographic information into prompts leads to a marginal but statistically significant improvement in LLMs' performance. We introduce a framework for evaluating the limitations of third-party annotations and call for refined annotation practices to accurately represent and model authors' private states.