# Capturing Reliable Fine-Grained Sentiment Associations by Crowdsourcing and Best-Worst Scaling

arXiv: https://arxiv.org/abs/1712.01741 · HTML: https://mlsanity.com/paper/1712.01741/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Dec 2017.

## Built on by (most cited first, 23 of 23)

- [AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZ](https://mlsanity.com/paper/2310.00367/) (2023)
- [Ruddit: Norms of Offensiveness for English Reddit Comments](https://mlsanity.com/paper/2106.05664/) (2021)
- [Best Practices in the Creation and Use of Emotion Lexicons](https://mlsanity.com/paper/2210.07206/) (2022)
- [BRIGHTER: BRIdging the Gap in Human-Annotated Textual Emotion Recognition Datasets for 28 Languages](https://mlsanity.com/paper/2502.11926/) (2025)
- [ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free Language Models](https://mlsanity.com/paper/2212.10474/) (2022)
- [TikZero: Zero-Shot Text-Guided Graphics Program Synthesis](https://mlsanity.com/paper/2503.11509/) (2025)
- [Akal Badi ya Bias: An Exploratory Study of Gender Bias in Hindi Language Technology](https://mlsanity.com/paper/2405.06346/) (2024)
- [Efficient Few-Shot Fine-Tuning for Opinion Summarization](https://mlsanity.com/paper/2205.02170/) (2022)
- [SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 13 Languages](https://mlsanity.com/paper/2402.08638/) (2024)
- [StreamHover: Livestream Transcript Summarization and Annotation](https://mlsanity.com/paper/2109.05160/) (2021)
- [What Makes Sentences Semantically Related: A Textual Relatedness Dataset and Empirical Study](https://mlsanity.com/paper/2110.04845/) (2021)
- [''Fifty Shades of Bias'': Normative Ratings of Gender Bias in GPT Generated English Text](https://mlsanity.com/paper/2310.17428/) (2023)
- [Unsupervised Abstractive Opinion Summarization by Generating Sentences with Tree-Structured Topic Guidance](https://mlsanity.com/paper/2106.08007/) (2021)
- [Susu Box or Piggy Bank: Assessing Cultural Commonsense Knowledge between Ghana and the U.S](https://mlsanity.com/paper/2410.16451/) (2024)
- [SemEval-2025 Task 11: Bridging the Gap in Text-Based Emotion Detection](https://mlsanity.com/paper/2503.07269/) (2025)
- [BabyStories: Can Reinforcement Learning Teach Baby Language Models to Write Better Stories?](https://mlsanity.com/paper/2310.16681/) (2023)
- [SemEval-2024 Task 1: Semantic Textual Relatedness for African and Asian Languages](https://mlsanity.com/paper/2403.18933/) (2024)
- [Quantifying Stereotypes in Language](https://mlsanity.com/paper/2401.15535/) (2024)
- ["splink" is happy and "phrouth" is scary: Emotion Intensity Analysis for Nonsense Words](https://mlsanity.com/paper/2202.12132/) (2022)
- [Dictionary-Assisted Supervised Contrastive Learning](https://mlsanity.com/paper/2210.15172/) (2022)
- [MasonTigers at SemEval-2024 Task 1: An Ensemble Approach for Semantic Textual Relatedness](https://mlsanity.com/paper/2403.14990/) (2024)
- [Privacy-Aware Crowd Labelling for Machine Learning Tasks](https://mlsanity.com/paper/2203.01373/) (2022)
- [An Attention-Based Model for Predicting Contextual Informativeness and Curriculum Learning Applications](https://mlsanity.com/paper/2204.09885/) (2022)

## Abstract (arXiv)

Access to word-sentiment associations is useful for many applications, including sentiment analysis, stance detection, and linguistic analysis. However, manually assigning fine-grained sentiment association scores to words has many challenges with respect to keeping annotations consistent. We apply the annotation technique of Best-Worst Scaling to obtain real-valued sentiment association scores for words and phrases in three different domains: general English, English Twitter, and Arabic Twitter. We show that on all three domains the ranking of words by sentiment remains remarkably consistent even when the annotation process is repeated with a different set of annotators. We also, for the first time, determine the minimum difference in sentiment association that is perceptible to native speakers of a language.