# QuMAB: Query-based Multi-Annotator Behavior Modeling with Reliability under Sparse Labels

arXiv: https://arxiv.org/abs/2507.17653 · HTML: https://mlsanity.com/paper/2507.17653/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jul 2025.

## Built on by (most cited first, 2 of 2)

- [STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systems](https://mlsanity.com/paper/2605.02122/) (2026)
- [A Unified Evaluation Framework for Multi-Annotator Tendency Learning](https://mlsanity.com/paper/2508.10393/) (2025)

## Abstract (arXiv)

Multi-annotator learning traditionally aggregates diverse annotations to approximate a single ground truth, treating disagreements as noise. However, this paradigm faces fundamental challenges: subjective tasks often lack absolute ground truth, and sparse annotation coverage makes aggregation statistically unreliable. We introduce a paradigm shift from sample-wise aggregation to annotator-wise behavior modeling. By treating annotator disagreements as valuable information rather than noise, modeling annotator-specific behavior patterns can reconstruct unlabeled data to reduce annotation cost, enhance aggregation reliability, and explain annotator decision behavior. To this end, we propose QuMAB (Query-based Multi-Annotator Behavior Pattern Learning), which uses light-weight queries to model individual annotators while capturing inter-annotator correlations as implicit regularization, preventing overfitting to sparse individual data while maintaining individualization and improving generalization, with a visualization of annotator focus regions offering an explainable analysis of behavior understanding. We contribute two large-scale datasets with dense per-annotator labels: STREET (4,300 labels/annotator) and AMER (average 3,118 labels/annotator), the first multimodal multi-annotator dataset. Extensive experiments demonstrate the superiority of our QuMAB in modeling individual annotators' behavior patterns, their utility for consensus prediction, and applicability under sparse annotations.