# AHEAD: Advancing Multi-Class Label Aggregation with Interpretable Cross-Annotator Modeling

arXiv: https://arxiv.org/abs/2607.18465 · HTML: https://mlsanity.com/paper/2607.18465/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 57 references that are in the map.
- First posted on arXiv 20 Jul 2026.

## Builds on (most cited first, 19 of 19)

- [Learning From Noisy Labels By Regularized Estimation Of Annotator Confusion](https://mlsanity.com/paper/1902.03680/) (2019)
- [Provably End-to-end Label-Noise Learning without Anchor Points](https://mlsanity.com/paper/2102.02400/) (2021)
- [Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generation](https://mlsanity.com/paper/2304.01816/) (2023)
- [Spectral Methods meet EM: A Provably Optimal Algorithm for Crowdsourcing](https://mlsanity.com/paper/1406.3824/) (2014)
- [Learning from Crowds by Modeling Common Confusions](https://mlsanity.com/paper/2012.13052/) (2020)
- [Crowdsourcing via Pairwise Co-occurrences: Identifiability and Algorithms](https://mlsanity.com/paper/1909.12325/) (2019)
- [A Comparative Study on Annotation Quality of Crowdsourcing and LLM via Label Aggregation](https://mlsanity.com/paper/2401.09760/) (2024)
- [Supernotes: Driving Consensus in Crowd-Sourced Fact-Checking](https://mlsanity.com/paper/2411.06116/) (2024)
- [Generative Contrastive Graph Learning for Recommendation](https://mlsanity.com/paper/2307.05100/) (2023)
- [SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLM](https://mlsanity.com/paper/2403.04735/) (2024)
- [Open Knowledge Enrichment for Long-tail Entities](https://mlsanity.com/paper/2002.06397/) (2020)
- [Deep Learning From Crowdsourced Labels: Coupled Cross-entropy Minimization, Identifiability, and Regularization](https://mlsanity.com/paper/2306.03288/) (2023)
- [Truth Inference at Scale: A Bayesian Model for Adjudicating Highly Redundant Crowd Annotations](https://mlsanity.com/paper/1902.08918/) (2019)
- [Coupled Confusion Correction: Learning from Crowds with Sparse Annotations](https://mlsanity.com/paper/2312.07331/) (2023)
- [Crowdsourcing via Annotator Co-occurrence Imputation and Provable Symmetric Nonnegative Matrix Factorization](https://mlsanity.com/paper/2106.07193/) (2021)
- [Improve Learning from Crowds via Generative Augmentation](https://mlsanity.com/paper/2107.10449/) (2021)
- [Cross-domain-aware Worker Selection with Training for Crowdsourced Annotation](https://mlsanity.com/paper/2406.06977/) (2024)
- [Bayesian Detector Combination for Object Detection with Crowdsourced Annotations](https://mlsanity.com/paper/2407.07958/) (2024)
- [The Viability of Crowdsourcing for RAG Evaluation](https://mlsanity.com/paper/2504.15689/) (2025)

## Abstract (arXiv)

Crowdsourced labeling provides valuable labeled data for domains across natural language processing, computer vision, and video. Label aggregation aims to infer latent true labels from noisy and biased annotations, with the key lying in annotator reliability estimation. Despite promising progress, existing approaches struggle with one real-world bottleneck: most individual annotators label only a small subset of tasks, making accurate annotator estimation highly intractable. In this paper, we focus on the considerably more challenging multi-class label aggregation and propose AHEAD (cross-Annotator learning and High-confidEnce Annotator-guideD label aggregation), a cross-annotator learning framework that advances annotator reliability estimation by leveraging the population-level data. Specifically, AHEAD first learns high-dimensional cross-annotator contexts via a graph neural network, deriving multi-view, complementary annotator embeddings by aggregating individual-level annotator features with contextual information. These embeddings are then decoded into interpretable annotator-specific confusion matrices to fit the observed labels. We formulate a composite objective incorporating high-confidence annotators to alleviate the unsupervised training issues faced by prior models. Experiments on 10 real-world datasets spanning NLP, CV, Video, and Audio show that AHEAD substantially improves label accuracy, increasing average accuracy from 68.75% to 73.23%, with gains of up to 14.9% in the best case. Meanwhile, scalability experiments on the largest dataset further demonstrate the overall superiority of our method.