mlsanityData to 7 Oct 2026

Efficient Online Scalar Annotation with Bounded Support

arXiv 1806.01170 ↗First posted 4 Jun 2018cs.CLcited by 27

Not on the map. Why?

Abstract (arXiv)

We describe a novel method for efficiently eliciting scalar annotations for dataset construction and system quality estimation by human judgments. We contrast direct assessment (annotators assign scores to items directly), online pairwise ranking aggregation (scores derive from annotator comparison of items), and a hybrid approach (EASL: Efficient Annotation of Scalar Labels) proposed here. Our proposal leads to increased correlation with ground truth, at far greater annotator efficiency, suggesting this strategy as an improved mechanism for dataset creation and manual system evaluation.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 8; 2023: 4; 2024: 3; 2025: 2; 2026: 4.

27 papers.

Sort:

By area:

  1. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  2. 2021A Recipe For Arbitrary Text Style Transfer with Large Language Modelscited by 107
  3. 2022SQuALITY: Building a Long-Document Summarization Dataset the Hard Waycited by 50
  4. 2022Grammatical Error Correction: A Survey of the State of the Artcited by 43
  5. 2021TellMeWhy: A Dataset for Answering Why-Questions in Narrativescited by 27
  6. 2021GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generationcited by 20
  7. 2022Czech Grammar Error Correction with a Large and Diverse Corpuscited by 17
  8. 2023Zero and Few-shot Semantic Parsing with Ambiguous Inputscited by 16
  9. 2022Recent Advances in Neural Text Generation: A Task-Agnostic Surveycited by 14
  10. 2022Ambiguous Images With Human Judgments for Robust Visual Event Classificationcited by 12
  11. 2023Using Natural Language Explanations to Rescale Human Judgmentscited by 11
  12. 2026Pearmut: Human Evaluation of Translation Made Trivialcited by 11
15 more
  1. 2022Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisonscited by 10
  2. 2025Measuring Scalar Constructs in Social Science with LLMscited by 8
  3. 2021How to Evaluate a Summarizer: Study Design and Statistical Analysis for Manual Linguistic Quality Evaluationcited by 7
  4. 2023Analyzing the Performance of GPT-3.5 and GPT-4 in Grammatical Error Correctioncited by 7
  5. 2022What to Read in a Contract? Party-Specific Summarization of Legal Obligations, Entitlements, and Prohibitionscited by 5
  6. 2021Input Augmentation Improves Constrained Beam Search for Neural Machine Translation: NTT at WAT 2021cited by 4
  7. 2023No Strong Feelings One Way or Another: Re-operationalizing Neutrality in Natural Language Inferencecited by 4
  8. 2024Better than Random: Reliable NLG Human Evaluation with Constrained Active Samplingcited by 4
  9. 2021On User Interfaces for Large-Scale Document-Level Human Evaluation of Machine Translation Outputscited by 3
  10. 2024Defeasible Visual Entailment: Benchmark, Evaluator, and Reward-Driven Optimizationcited by 3
  11. 2024Baby Bear: Seeking a Just Right Rating Scale for Scalar Annotationscited by 1
  12. 2026PEAR: Pairwise Evaluation for Automatic Relative Scoring in Machine Translationcited by 1
  13. 2026Contrastive ESA: Human Evaluation of Multiple Translations at Oncecited by 1
  14. 2025Time Is Effort: Estimating Human Post-Editing Time for Grammar Error Correction Tool Evaluationcited by 0
  15. 2026Dynamically Allocating Evaluation Effort for Model Rankingcited by 0