mlsanityData to 7 Oct 2026

Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotation

arXiv 1712.01765 ↗First posted 5 Dec 2017cs.CLcited by 77

Not on the map. Why?

Abstract (arXiv)

Rating scales are a widely used method for data annotation; however, they present several challenges, such as difficulty in maintaining inter- and intra-annotator consistency. Best-worst scaling (BWS) is an alternative method of annotation that is claimed to produce high-quality annotations while keeping the required number of annotations similar to that of rating scales. However, the veracity of this claim has never been systematically established. Here for the first time, we set up an experiment that directly compares the rating scale method with BWS. We show that with the same total number of annotations, BWS produces significantly more reliable results than the rating scale.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Dec 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 18; 2023: 13; 2024: 14; 2025: 8; 2026: 15.

77 papers.

Sort:

By area:

  1. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  2. 2023AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZcited by 74
  3. 2022Detoxifying Text with MaRCo: Controllable Revision with Experts and Anti-Expertscited by 36
  4. 2022Emotion Intensity and its Control for Emotional Voice Conversioncited by 35
  5. 2023Faithfulness-Aware Decoding Strategies for Abstractive Summarizationcited by 31
  6. 2024DRT: Deep Reasoning Translation via Long Chain-of-Thoughtcited by 28
  7. 2021Cross-Lingual Abstractive Summarization with Limited Parallel Resourcescited by 27
  8. 2022Best Practices in the Creation and Use of Emotion Lexiconscited by 27
  9. 2021Ruddit: Norms of Offensiveness for English Reddit Commentscited by 26
  10. 2022ClidSum: A Benchmark Dataset for Cross-Lingual Dialogue Summarizationcited by 25
  11. 2022ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free Language Modelscited by 25
  12. 2023Generating Summaries with Controllable Readability Levelscited by 24
38 more
  1. 2024How Good Are LLMs for Literary Translation, Really? Literary Translation Evaluation with Humans and LLMscited by 23
  2. 2023Consistency is Key: Disentangling Label Variation in Natural Language Processing with Intra-Annotator Agreementcited by 22
  3. 2025TikZero: Zero-Shot Text-Guided Graphics Program Synthesiscited by 21
  4. 2021Self-Supervised Multimodal Opinion Summarizationcited by 18
  5. 2024Akal Badi ya Bias: An Exploratory Study of Gender Bias in Hindi Language Technologycited by 17
  6. 2024SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 13 Languagescited by 16
  7. 2021Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summariescited by 15
  8. 2021What Makes Sentences Semantically Related: A Textual Relatedness Dataset and Empirical Studycited by 15
  9. 2022Towards Explainable Evaluation Metrics for Natural Language Generationcited by 15
  10. 2023''Fifty Shades of Bias'': Normative Ratings of Gender Bias in GPT Generated English Textcited by 15
  11. 2022Recent Advances in Neural Text Generation: A Task-Agnostic Surveycited by 14
  12. 2022PSP: Pre-trained Soft Prompts for Few-Shot Abstractive Summarizationcited by 14
  13. 2025ExTrans: Multilingual Deep Reasoning Translation via Exemplar-Enhanced Reinforcement Learningcited by 11
  14. 2023Less is More: Mitigate Spurious Correlations for Open-Domain Dialogue Response Generation Models by Causal Discoverycited by 10
  15. 2023Large-Scale and Multi-Perspective Opinion Summarization with Diverse Review Subsetscited by 10
  16. 2022Template-based Abstractive Microblog Opinion Summarisationcited by 9
  17. 2022AARGH! End-to-end Retrieval-Generation for Task-Oriented Dialogcited by 8
  18. 2024Understanding Fine-grained Distortions in Reports of Scientific Findingscited by 8
  19. 2024QAPyramid: Fine-grained Evaluation of Content Selection for Text Summarizationcited by 8
  20. 2021How to Evaluate a Summarizer: Study Design and Statistical Analysis for Manual Linguistic Quality Evaluationcited by 7
  21. 2021Mitigating Media Bias through Neutral Article Generationcited by 7
  22. 2024SemEval-2024 Task 1: Semantic Textual Relatedness for African and Asian Languagescited by 7
  23. 2024AGENT-CQ: Automatic Generation and Evaluation of Clarifying Questions for Conversational Search with LLMscited by 7
  24. 2023Don't Take This Out of Context! On the Need for Contextual Models and Evaluations for Stylistic Rewritingcited by 6
  25. 2022Counterfactual Recipe Generation: Exploring Compositional Generalization in a Realistic Scenariocited by 5
  26. 2022What to Read in a Contract? Party-Specific Summarization of Legal Obligations, Entitlements, and Prohibitionscited by 5
  27. 2024CausalScore: An Automatic Reference-Free Metric for Assessing Response Relevance in Open-Domain Dialogue Systemscited by 5
  28. 2025AV-Dialog: Spoken Dialogue Models with Audio-Visual Inputcited by 5
  29. 2022Which one is more toxic? Findings from Jigsaw Rate Severity of Toxic Commentscited by 4
  30. 2024Chitchat as Interference: Adding User Backstories to Task-Oriented Dialoguescited by 4
  31. 2022ClueGraphSum: Let Key Clues Guide the Cross-Lingual Abstractive Summarizationcited by 3
  32. 2024Quantifying Stereotypes in Languagecited by 3
  33. 2024NLU-STR at SemEval-2024 Task 1: Generative-based Augmentation and Encoder-based Scoring for Semantic Textual Relatednesscited by 3
  34. 2025Scaling Multi-Document Event Summarization: Evaluating Compression vs. Full-Text Approachescited by 3
  35. 2024"You are an expert annotator": Automatic Best-Worst-Scaling Annotations for Emotion Intensity Modelingcited by 2
  36. 2026The Voice Behind the Words: Quantifying Intersectional Bias in SpeechLLMscited by 2
  37. 2022Predicting pairwise preferences between TTS audio stimuli using parallel ratings data and anti-symmetric twin neural networkscited by 1
  38. 2023Background Summarization of Event Timelinescited by 1

50 most cited of 77 All 77 (JSON)