# Efficient Annotator Reliability Assessment with EffiARA

arXiv: https://arxiv.org/abs/2504.00589 · HTML: https://mlsanity.com/paper/2504.00589/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 41 references that are in the map.
- First posted on arXiv 1 Apr 2025.

## Builds on (most cited first, 6 of 6)

- [CrowdTruth 2.0: Quality Metrics for Crowdsourcing with Disagreement](https://mlsanity.com/paper/1808.06080/) (2018)
- [Learning from Crowds with Crowd-Kit](https://mlsanity.com/paper/2109.08584/) (2021)
- [Don't Waste a Single Annotation: Improving Single-Label Classifiers Through Soft Labels](https://mlsanity.com/paper/2311.05265/) (2023)
- [Knowing Your Annotator: Rapidly Testing the Reliability of Affect Annotation](https://mlsanity.com/paper/2308.16029/) (2023)
- [Efficient Annotator Reliability Assessment and Sample Weighting for Knowledge-Based Misinformation Detection on Social Media](https://mlsanity.com/paper/2410.14515/) (2024)
- [A Dataset for Analysing News Framing in Chinese Media](https://mlsanity.com/paper/2503.04439/) (2025)

## Built on by (most cited first, 1 of 1)

- [What Makes a Sale? Simulating End-to-End Seller--Buyer Retail Dynamics with LLM Agents](https://mlsanity.com/paper/2604.04468/) (2026)

## Abstract (arXiv)

Data annotation is an essential component of the machine learning pipeline; it is also a costly and time-consuming process. With the introduction of transformer-based models, annotation at the document level is increasingly popular; however, there is no standard framework for structuring such tasks. The EffiARA annotation framework is, to our knowledge, the first project to support the whole annotation pipeline, from understanding the resources required for an annotation task to compiling the annotated dataset and gaining insights into the reliability of individual annotators as well as the dataset as a whole. The framework's efficacy is supported by two previous studies: one improving classification performance through annotator-reliability-based soft-label aggregation and sample weighting, and the other increasing the overall agreement among annotators through removing identifying and replacing an unreliable annotator. This work introduces the EffiARA Python package and its accompanying webtool, which provides an accessible graphical user interface for the system. We open-source the EffiARA Python package at https://github.com/MiniEggz/EffiARA and the webtool is publicly accessible at https://effiara.gate.ac.uk.