# Crowdsourcing a Word-Emotion Association Lexicon

arXiv: https://arxiv.org/abs/1308.6297 · HTML: https://mlsanity.com/paper/1308.6297/

## Key facts

- Cited by 196 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Aug 2013.

## Built on by (most cited first, 30 of 196)

- [A Survey on Gender Bias in Natural Language Processing](https://mlsanity.com/paper/2112.14168/) (2021)
- [EmoBank: Studying the Impact of Annotation Perspective and Representation Format on Dimensional Emotion Analysis](https://mlsanity.com/paper/2205.01996/) (2022)
- [NaijaSenti: A Nigerian Twitter Sentiment Corpus for Multilingual Sentiment Analysis](https://mlsanity.com/paper/2201.08277/) (2022)
- [Mind the Sim2Real Gap in User Simulation for Agentic Tasks](https://mlsanity.com/paper/2603.11245/) (2026)
- [Towards Interpretable Mental Health Analysis with Large Language Models](https://mlsanity.com/paper/2304.03347/) (2023)
- [Dimensional Modeling of Emotions in Text with Appraisal Theories: Corpus Creation, Annotation Reliability, and Prediction](https://mlsanity.com/paper/2206.05238/) (2022)
- [Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes](https://mlsanity.com/paper/2109.08828/) (2021)
- [SpanEmo: Casting Multi-label Emotion Classification as Span-prediction](https://mlsanity.com/paper/2101.10038/) (2021)
- [The Shrinking Landscape of Linguistic Diversity in the Age of Large Language Models](https://mlsanity.com/paper/2502.11266/) (2025)
- [Best Practices in the Creation and Use of Emotion Lexicons](https://mlsanity.com/paper/2210.07206/) (2022)
- [Integrating Pattern- and Fact-based Fake News Detection via Model Preference Learning](https://mlsanity.com/paper/2109.11333/) (2021)
- [Domain-specific Continued Pretraining of Language Models for Capturing Long Context in Mental Health](https://mlsanity.com/paper/2304.10447/) (2023)
- [Ethics Sheet for Automatic Emotion Recognition and Sentiment Analysis](https://mlsanity.com/paper/2109.08256/) (2021)
- [Emotion Detection for Misinformation: A Review](https://mlsanity.com/paper/2311.00671/) (2023)
- [Emotion Dynamics in Movie Dialogues](https://mlsanity.com/paper/2103.01345/) (2021)
- [On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies](https://mlsanity.com/paper/2104.05694/) (2021)
- [Evaluating the Capabilities of Large Language Models for Multi-label Emotion Understanding](https://mlsanity.com/paper/2412.17837/) (2024)
- [Evaluating Emotion Arcs Across Languages: Bridging the Global Divide in Sentiment Analysis](https://mlsanity.com/paper/2306.02213/) (2023)
- [''Fifty Shades of Bias'': Normative Ratings of Gender Bias in GPT Generated English Text](https://mlsanity.com/paper/2310.17428/) (2023)
- [HeBERT & HebEMO: a Hebrew BERT Model and a Tool for Polarity Analysis and Emotion Recognition](https://mlsanity.com/paper/2102.01909/) (2021)
- [An Exploratory Analysis of the Relation Between Offensive Language and Mental Health](https://mlsanity.com/paper/2105.14888/) (2021)
- [MetaHate: A Dataset for Unifying Efforts on Hate Speech Detection](https://mlsanity.com/paper/2401.06526/) (2024)
- [The Project Dialogism Novel Corpus: A Dataset for Quotation Attribution in Literary Texts](https://mlsanity.com/paper/2204.05836/) (2022)
- [Natural Language Inference Prompts for Zero-shot Emotion Classification in Text across Corpora](https://mlsanity.com/paper/2209.06701/) (2022)
- [Exploring the Efficacy of Pre-trained Checkpoints in Text-to-Music Generation Task](https://mlsanity.com/paper/2211.11216/) (2022)
- [Emotion fusion for mental illness detection from social media: A survey](https://mlsanity.com/paper/2304.09493/) (2023)
- [Automatic Identification and Classification of Bragging in Social Media](https://mlsanity.com/paper/2203.05840/) (2022)
- [Calibration of Transformer-based Models for Identifying Stress and Depression in Social Media](https://mlsanity.com/paper/2305.16797/) (2023)
- [The Emotion Dynamics of Literary Novels](https://mlsanity.com/paper/2403.02474/) (2024)
- [Still Not Quite There! Evaluating Large Language Models for Comorbid Mental Health Diagnosis](https://mlsanity.com/paper/2410.03908/) (2024)

## Abstract (arXiv)

Even though considerable attention has been given to the polarity of words (positive and negative) and the creation of large polarity lexicons, research in emotion analysis has had to rely on limited and small emotion lexicons. In this paper we show how the combined strength and wisdom of the crowds can be used to generate a large, high-quality, word-emotion and word-polarity association lexicon quickly and inexpensively. We enumerate the challenges in emotion annotation in a crowdsourcing scenario and propose solutions to address them. Most notably, in addition to questions about emotions associated with terms, we show how the inclusion of a word choice question can discourage malicious data entry, help identify instances where the annotator may not be familiar with the target term (allowing us to reject such annotations), and help obtain annotations at sense level (rather than at word level). We conducted experiments on how to formulate the emotion-annotation questions, and show that asking if a term is associated with an emotion leads to markedly higher inter-annotator agreement than that obtained by asking if a term evokes an emotion.