mlsanityData to 14 Oct 2026

Consistency is Key: Disentangling Label Variation in Natural Language Processing with Intra-Annotator Agreement

arXiv 2301.10684 ↗First posted 25 Jan 2023cs.CLcited by 23

Not on the map. Why?

Abstract (arXiv)

We commonly use agreement measures to assess the utility of judgements made by human annotators in Natural Language Processing (NLP) tasks. While inter-annotator agreement is frequently used as an indication of label reliability by measuring consistency between annotators, we argue for the additional use of intra-annotator agreement to measure label stability (and annotator consistency) over time. However, in a systematic review, we find that the latter is rarely reported in this field. Calculating these measures can act as important quality control and could provide insights into why annotators disagree. We conduct exploratory annotation experiments to investigate the relationships between these measures and perceptions of subjectivity and ambiguity in text items, finding that annotators provide inconsistent responses around 25% of the time across four different NLP tasks.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 21 of its 100 references that are in the map. First posted on arXiv 25 Jan 2023.

Builds on

21 of 100 references.

References · 21

  1. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  2. 2021MultiVerS: Improving scientific claim verification with weak supervision and full-document contextcited by 143
  3. 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 139
  4. 2021Toward a Perspectivist Turn in Ground Truthing for Predictive Computingcited by 130
  5. 2020On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translationcited by 127
  6. 2017Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotationcited by 79
  7. 2021Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' Disagreementcited by 72
  8. 2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learningcited by 57
  9. 2021Beyond Fair Pay: Ethical Implications of NLP Crowdsourcingcited by 53
  10. 2021ConvAbuse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational AIcited by 46
  11. 2022Theory-Grounded Measurement of U.S. Social Stereotypes in English Language Modelscited by 35
  12. 2019Why Does a Visual Question Have Different Answers?cited by 34
9 more
  1. 2021Preregistering NLP Researchcited by 25
  2. 2021SemEval-2021 Task 11: NLPContributionGraph -- Structuring Scholarly NLP Contributions for a Research Knowledge Graphcited by 16
  3. 2022StereoKG: Data-Driven Knowledge Graph Construction for Cultural Knowledge and Stereotypescited by 14
  4. 2020Shallow Discourse Annotation for Chinese TED Talkscited by 13
  5. 2019Analysis of Automatic Annotation Suggestions for Hard Discourse-Level Tasks in Expert Domainscited by 11
  6. 2020Correct Me If You Can: Learning from Error Corrections and Markingscited by 11
  7. 2016Query-Focused Opinion Summarization for User-Generated Contentcited by 7
  8. 2022Exploiting Social Media Content for Self-Supervised Style Transfercited by 2
  9. 2021SuperSim: a test set for word similarity and relatedness in Swedishcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 6; 2025: 6; 2026: 7.

23 papers.

Sort:

By area:

  1. 2024The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labelscited by 38
  2. 2025Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworkscited by 38
  3. 2025Value Profiles for Encoding Human Variationcited by 29
  4. 2024Conformalized Credal Set Predictorscited by 21
  5. 2023What about em? How Commercial Machine Translation Fails to Handle (Neo-)Pronounscited by 19
  6. 2026Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLPcited by 13
  7. 2026Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluationcited by 13
  8. 2024Voices in a Crowd: Searching for Clusters of Unique Perspectivescited by 9
  9. 2023Evaluating AI systems under uncertain ground truth: a case study in dermatologycited by 7
  10. 2023Architectural Sweet Spots for Modeling Human Label Variation by the Example of Argument Quality: It's Best to Relate Perspectives!cited by 5
  11. 2023A Taxonomy of Rater Disagreements: Surveying Challenges & Opportunities from the Perspective of Annotating Online Toxicitycited by 4
  12. 2025Perspectives in Play: A Multi-Perspective Approach for More Inclusive NLP Systemscited by 4
11 more
  1. 2025Optimal Conformal Prediction under Epistemic Uncertaintycited by 3
  2. 2025eMotions: A Large-Scale Dataset and Audio-Visual Fusion Network for Emotion Analysis in Short-form Videoscited by 2
  3. 2024Learning to Explore and Select for Coverage-Conditioned Retrieval-Augmented Generationcited by 1
  4. 2026P1SCO: Social Dimensions from a Perspectivist Lenscited by 1
  5. 2026Contrastive ESA: Human Evaluation of Multiple Translations at Oncecited by 1
  6. 2024Malaysian English News Decoded: A Linguistic Resource for Named Entity and Relation Extractioncited by 0
  7. 2024Linguistic Fingerprint in Transformer Models: How Language Variation Influences Parameter Selection in Irony Detectioncited by 0
  8. 2025From Ground Trust to Truth: Disparities in Offensive Language Judgments on Contemporary Korean Political Discoursecited by 0
  9. 2026Exploiting the English Grammar Profile for L2 grammatical analysis with LLMscited by 0
  10. 2026Not All Subjectivity Is the Same! Defining Desiderata for the Evaluation of Subjectivity in NLPcited by 0
  11. 2026Boundaries Agree, Labels Do Not: Intra-Annotator Dynamics as a Kind of Training Datacited by 0