# Cross-replication Reliability -- An Empirical Approach to Interpreting Inter-rater Reliability

arXiv: https://arxiv.org/abs/2106.07393 · HTML: https://mlsanity.com/paper/2106.07393/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 50 references that are in the map.
- First posted on arXiv 11 Jun 2021.

## Builds on (most cited first, 3 of 3)

- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Identifying Statistical Bias in Dataset Replication](https://mlsanity.com/paper/2005.09619/) (2020)

## Built on by (most cited first, 30 of 31)

- [A Toolbox for Surfacing Health Equity Harms and Biases in Large Language Models](https://mlsanity.com/paper/2403.12025/) (2024)
- [TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Models](https://mlsanity.com/paper/2405.18027/) (2024)
- [Measuring and Improving Semantic Diversity of Dialogue Generation](https://mlsanity.com/paper/2210.05725/) (2022)
- [GRASP: A Disagreement Analysis Framework to Assess Group Associations in Perspectives](https://mlsanity.com/paper/2311.05074/) (2023)
- [Dynamic Multi-Reward Weighting for Multi-Style Controllable Generation](https://mlsanity.com/paper/2402.14146/) (2024)
- [Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluation](https://mlsanity.com/paper/2603.06865/) (2026)
- [D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation](https://mlsanity.com/paper/2404.10857/) (2024)
- [Whose View of Safety? A Deep DIVE Dataset for Pluralistic Alignment of Text-to-Image Models](https://mlsanity.com/paper/2507.13383/) (2025)
- [Rater Cohesion and Quality from a Vicarious Perspective](https://mlsanity.com/paper/2408.08411/) (2024)
- [k-Rater Reliability: The Correct Unit of Reliability for Aggregated Human Annotations](https://mlsanity.com/paper/2203.12913/) (2022)
- [What Is The Political Content in LLMs' Pre- and Post-Training Data?](https://mlsanity.com/paper/2509.22367/) (2025)
- [Modeling subjectivity (by Mimicking Annotator Annotation) in toxic comment identification across diverse communities](https://mlsanity.com/paper/2311.00203/) (2023)
- [Collect, Measure, Repeat: Reliability Factors for Responsible AI Data Collection](https://mlsanity.com/paper/2308.12885/) (2023)
- [Evolving Domain Adaptation of Pretrained Language Models for Text Classification](https://mlsanity.com/paper/2311.09661/) (2023)
- [Mitigating Unhelpfulness in Emotional Support Conversations with Multifaceted AI Feedback](https://mlsanity.com/paper/2401.05928/) (2024)
- [Undesirable Biases in NLP: Addressing Challenges of Measurement](https://mlsanity.com/paper/2211.13709/) (2022)
- [Localizing and Mitigating Errors in Long-form Question Answering](https://mlsanity.com/paper/2407.11930/) (2024)
- [Multimodal Fusion with LLMs for Engagement Prediction in Natural Conversation](https://mlsanity.com/paper/2409.09135/) (2024)
- [RIGOURATE: Quantifying Scientific Exaggeration with Evidence-Aligned Claim Evaluation](https://mlsanity.com/paper/2601.04350/) (2026)
- [Agent Benchmarks Fail Public Sector Requirements](https://mlsanity.com/paper/2601.20617/) (2026)
- [Student's t-Distribution: On Measuring the Inter-Rater Reliability When the Observations are Scarce](https://mlsanity.com/paper/2303.04526/) (2023)
- [How We Define Harm Impacts Data Annotations: Explaining How Annotators Distinguish Hateful, Offensive, and Toxic Comments](https://mlsanity.com/paper/2309.15827/) (2023)
- [Beyond the Explicit: A Bilingual Dataset for Dehumanization Detection in Social Media](https://mlsanity.com/paper/2510.18582/) (2025)
- [The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation](https://mlsanity.com/paper/2602.11318/) (2026)
- [Can LLMs Evaluate What They Cannot Annotate? Revisiting LLM Reliability in Hate Speech Detection](https://mlsanity.com/paper/2512.09662/) (2025)
- [Surfacing the Unsaid: CUE-Bench for Affective Stance in Chinese Discourse](https://mlsanity.com/paper/2608.10810/) (2026)
- [Tracing How Annotators Think: Augmenting Preference Judgments with Reading Processes](https://mlsanity.com/paper/2511.21912/) (2025)
- [Are we chasing ghosts? Quantifying unattributable polarization, and attributing the rest to annotator groups](https://mlsanity.com/paper/2602.06055/) (2026)
- [Effective Performance Measurement: Challenges and Opportunities in KPI Extraction from Earnings Calls](https://mlsanity.com/paper/2605.03147/) (2026)
- [TIDES: A Longitudinal Bilingual Dataset for Modeling Multi-Party Social Dynamics](https://mlsanity.com/paper/2608.01724/) (2026)

## Abstract (arXiv)

We present a new approach to interpreting IRR that is empirical and contextualized. It is based upon benchmarking IRR against baseline measures in a replication, one of which is a novel cross-replication reliability (xRR) measure based on Cohen's kappa. We call this approach the xRR framework. We opensource a replication dataset of 4 million human judgements of facial expressions and analyze it with the proposed framework. We argue this framework can be used to measure the quality of crowdsourced datasets.