# Student's t-Distribution: On Measuring the Inter-Rater Reliability When the Observations are Scarce

arXiv: https://arxiv.org/abs/2303.04526 · HTML: https://mlsanity.com/paper/2303.04526/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 41 references that are in the map.
- First posted on arXiv 8 Mar 2023.

## Builds on (most cited first, 9 of 9)

- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [Re-evaluating Evaluation in Text Summarization](https://mlsanity.com/paper/2010.07100/) (2020)
- [Cross-replication Reliability -- An Empirical Approach to Interpreting Inter-rater Reliability](https://mlsanity.com/paper/2106.07393/) (2021)
- [Translation Quality Assessment: A Brief Survey on Manual and Automatic Methods](https://mlsanity.com/paper/2105.03311/) (2021)
- [HOPE: A Task-Oriented and Human-Centric Evaluation Framework Using Professional Post-Editing Towards More Effective MT Evaluation](https://mlsanity.com/paper/2112.13833/) (2021)
- [Measuring Uncertainty in Translation Quality Evaluation (TQE)](https://mlsanity.com/paper/2111.07699/) (2021)
- [Meta-Evaluation of Translation Evaluation Methods: a systematic up-to-date overview](https://mlsanity.com/paper/1605.04515/) (2016)
- [An Overview on Machine Translation Evaluation](https://mlsanity.com/paper/2202.11027/) (2022)
- [From web crawled text to project descriptions: automatic summarizing of social innovation projects](https://mlsanity.com/paper/1905.09086/) (2019)

## Built on by (most cited first, 2 of 2)

- [The Multi-Range Theory of Translation Quality Measurement: MQM scoring models and Statistical Quality Control](https://mlsanity.com/paper/2405.16969/) (2024)
- [AmpleHate: Amplifying the Attention for Versatile Implicit Hate Detection](https://mlsanity.com/paper/2505.19528/) (2025)

## Abstract (arXiv)

In natural language processing (NLP) we always rely on human judgement as the golden quality evaluation method. However, there has been an ongoing debate on how to better evaluate inter-rater reliability (IRR) levels for certain evaluation tasks, such as translation quality evaluation (TQE), especially when the data samples (observations) are very scarce. In this work, we first introduce the study on how to estimate the confidence interval for the measurement value when only one data (evaluation) point is available. Then, this leads to our example with two human-generated observational scores, for which, we introduce ``Student's textitt-Distribution'' method and explain how to use it to measure the IRR score using only these two data points, as well as the confidence intervals (CIs) of the quality evaluation. We give quantitative analysis on how the evaluation confidence can be greatly improved by introducing more observations, even if only one extra observation. We encourage researchers to report their IRR scores in all possible means, e.g. using Student's textitt-Distribution method whenever possible; thus making the NLP evaluation more meaningful, transparent, and trustworthy. This textitt-Distribution method can be also used outside of NLP fields to measure IRR level for trustworthy evaluation of experimental investigations, whenever the observational data is scarce. Keywords: Inter-Rater Reliability (IRR); Scarce Observations; Confidence Intervals (CIs); Natural Language Processing (NLP); Translation Quality Evaluation (TQE); Student's textitt-Distribution