mlsanityData to 14 Oct 2026

Cross-replication Reliability -- An Empirical Approach to Interpreting Inter-rater Reliability

arXiv 2106.07393 ↗First posted 11 Jun 2021stat.AP cs.AI cs.SIcited by 31

Not on the map. Why?

Abstract (arXiv)

We present a new approach to interpreting IRR that is empirical and contextualized. It is based upon benchmarking IRR against baseline measures in a replication, one of which is a novel cross-replication reliability (xRR) measure based on Cohen's kappa. We call this approach the xRR framework. We opensource a replication dataset of 4 million human judgements of facial expressions and analyze it with the proposed framework. We argue this framework can be used to measure the quality of crowdsourced datasets.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 3 of its 50 references that are in the map. First posted on arXiv 11 Jun 2021.

Builds on

3 of 50 references.

References · 3

  1. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  2. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  3. 2020Identifying Statistical Bias in Dataset Replicationcited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 6; 2024: 8; 2025: 5; 2026: 9.

31 papers.

Sort:

By area:

  1. 2024A Toolbox for Surfacing Health Equity Harms and Biases in Large Language Modelscited by 45
  2. 2024TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Modelscited by 22
  3. 2022Measuring and Improving Semantic Diversity of Dialogue Generationcited by 16
  4. 2023GRASP: A Disagreement Analysis Framework to Assess Group Associations in Perspectivescited by 16
  5. 2024Dynamic Multi-Reward Weighting for Multi-Style Controllable Generationcited by 13
  6. 2026Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluationcited by 13
  7. 2024D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluationcited by 11
  8. 2025Whose View of Safety? A Deep DIVE Dataset for Pluralistic Alignment of Text-to-Image Modelscited by 11
  9. 2024Rater Cohesion and Quality from a Vicarious Perspectivecited by 8
  10. 2022k-Rater Reliability: The Correct Unit of Reliability for Aggregated Human Annotationscited by 7
  11. 2025What Is The Political Content in LLMs' Pre- and Post-Training Data?cited by 7
  12. 2023Modeling subjectivity (by Mimicking Annotator Annotation) in toxic comment identification across diverse communitiescited by 6
19 more
  1. 2023Collect, Measure, Repeat: Reliability Factors for Responsible AI Data Collectioncited by 5
  2. 2023Evolving Domain Adaptation of Pretrained Language Models for Text Classificationcited by 5
  3. 2024Mitigating Unhelpfulness in Emotional Support Conversations with Multifaceted AI Feedbackcited by 5
  4. 2022Undesirable Biases in NLP: Addressing Challenges of Measurementcited by 4
  5. 2024Localizing and Mitigating Errors in Long-form Question Answeringcited by 4
  6. 2024Multimodal Fusion with LLMs for Engagement Prediction in Natural Conversationcited by 3
  7. 2026RIGOURATE: Quantifying Scientific Exaggeration with Evidence-Aligned Claim Evaluationcited by 3
  8. 2026Agent Benchmarks Fail Public Sector Requirementscited by 3
  9. 2023Student's t-Distribution: On Measuring the Inter-Rater Reliability When the Observations are Scarcecited by 2
  10. 2023How We Define Harm Impacts Data Annotations: Explaining How Annotators Distinguish Hateful, Offensive, and Toxic Commentscited by 2
  11. 2025Beyond the Explicit: A Bilingual Dataset for Dehumanization Detection in Social Mediacited by 2
  12. 2026The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotationcited by 2
  13. 2025Can LLMs Evaluate What They Cannot Annotate? Revisiting LLM Reliability in Hate Speech Detectioncited by 1
  14. 2026Surfacing the Unsaid: CUE-Bench for Affective Stance in Chinese Discoursecited by 1
  15. 2025Tracing How Annotators Think: Augmenting Preference Judgments with Reading Processescited by 0
  16. 2026Are we chasing ghosts? Quantifying unattributable polarization, and attributing the rest to annotator groupscited by 0
  17. 2026Effective Performance Measurement: Challenges and Opportunities in KPI Extraction from Earnings Callscited by 0
  18. 2026TIDES: A Longitudinal Bilingual Dataset for Modeling Multi-Party Social Dynamicscited by 0
  19. 2026Beyond Majority Vote: Multi-Perspective Adjudication for Medical Hallucination Detectioncited by 0