mlsanityData to 7 Oct 2026

Let Guidelines Guide You: A Prescriptive Guideline-Centered Data Annotation Methodology

arXiv 2406.14099 ↗First posted 20 Jun 2024cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

We introduce the Guideline-Centered Annotation Methodology (GCAM), a novel data annotation methodology designed to report the annotation guidelines associated with each data sample. Our approach addresses three key limitations of the standard prescriptive annotation methodology by reducing the information loss during annotation and ensuring adherence to guidelines. Furthermore, GCAM enables the efficient reuse of annotated data across multiple tasks. We evaluate GCAM in two ways: (i) through a human annotation study and (ii) an experimental evaluation with several machine learning models. Our results highlight the advantages of GCAM from multiple perspectives, demonstrating its potential to improve annotation quality and error analysis.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 52 of its 94 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

52 of 94 references.

References · 52

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2023Mistral 7Bcited by 3,065
  3. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  5. 2018Datasheets for Datasetscited by 1,371
  6. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  7. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  8. 2022A Survey on In-context Learningcited by 717
  9. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  10. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
  11. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 414
  12. 2020Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)cited by 318
38 more
  1. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 306
  2. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 302
  3. 2020Data and its (dis)contents: A survey of dataset development and use in machine learning researchcited by 259
  4. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 258
  5. 2019Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systemscited by 203
  6. 2023Data-centric Artificial Intelligence: A Surveycited by 202
  7. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 201
  8. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 171
  9. 2022Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AIcited by 169
  10. 2021MultiVerS: Improving scientific claim verification with weak supervision and full-document contextcited by 140
  11. 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 133
  12. 2021Toward a Perspectivist Turn in Ground Truthing for Predictive Computingcited by 129
  13. 2024To Believe or Not to Believe Your LLMcited by 120
  14. 2023GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extractioncited by 116
  15. 2020LEGAL-BERT: The Muppets straight out of Law Schoolcited by 111
  16. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  17. 2020Stance Detection on Social Media: State of the Art and Trendscited by 93
  18. 2023CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotationcited by 93
  19. 2023NLPositionality: Characterizing Design Biases of Datasets and Modelscited by 91
  20. 2023When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Taskscited by 89
  21. 2023UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognitioncited by 70
  22. 2018Joint Embedding of Words and Labels for Text Classificationcited by 69
  23. 2022Handling Bias in Toxic Speech Detection: A Surveycited by 60
  24. 2022CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotationcited by 59
  25. 2021Mitigating Dataset Harms Requires Stewardship: Lessons from 1000 Paperscited by 49
  26. 2020Detecting Stance in Media on Global Warmingcited by 48
  27. 2023SemEval-2023 Task 10: Explainable Detection of Online Sexismcited by 48
  28. 2022Annotation Error Detection: Analyzing the Past and Present for a More Coherent Futurecited by 45
  29. 2023The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographicscited by 45
  30. 2022Prompting Language Models for Linguistic Structurecited by 39
  31. 2023Analyzing Dataset Annotation Quality Management in the Wildcited by 39
  32. 2018GILE: A Generalized Input-Label Embedding for Text Classificationcited by 22
  33. 2022AnnoBERT: Effectively Representing Multiple Annotators' Label Choices to Improve Hate Speech Detectioncited by 12
  34. 2022Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarizationcited by 10
  35. 2023ActiveAED: A Human in the Loop Improves Annotation Error Detectioncited by 9
  36. 2024"Define Your Terms" : Enhancing Efficient Offensive Speech Classification with Definitioncited by 7
  37. 2023Antisemitic Messages? A Guide to High-Quality Annotation and a Labeled Dataset of Tweetscited by 6
  38. 2023A Corpus for Sentence-level Subjectivity Detection on English News Articlescited by 6

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025What Has Been Lost with Synthetic Evaluation?cited by 8
  2. 2026Beyond Black-Box Labels: Interpretable Criteria for Diagnosing Subjective NLP Taskscited by 0
  3. 2026Automatically Evolving Prompt Guidelines for Task-Specific Optimizationcited by 0