# What is the ground truth? Reliability of multi-annotator data for audio tagging

arXiv: https://arxiv.org/abs/2104.04214 · HTML: https://mlsanity.com/paper/2104.04214/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Apr 2021.

## Built on by (most cited first, 29 of 29)

- [Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models](https://mlsanity.com/paper/2301.12661/) (2023)
- [Kimi-Audio Technical Report](https://mlsanity.com/paper/2504.18425/) (2025)
- [Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation](https://mlsanity.com/paper/2305.18474/) (2023)
- [JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronization](https://mlsanity.com/paper/2503.23377/) (2025)
- [Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generation](https://mlsanity.com/paper/2401.01044/) (2024)
- [Audio Retrieval with WavText5K and CLAP Training](https://mlsanity.com/paper/2209.14275/) (2022)
- [JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generation](https://mlsanity.com/paper/2602.19163/) (2026)
- [AudioLCM: Text-to-Audio Generation with Latent Consistency Models](https://mlsanity.com/paper/2406.00356/) (2024)
- [The iNaturalist Sounds Dataset](https://mlsanity.com/paper/2506.00343/) (2025)
- [Object-AVEdit: An Object-level Audio-Visual Editing Model](https://mlsanity.com/paper/2510.00050/) (2025)
- [AudioEditor: A Training-Free Diffusion-Based Audio Editing Framework](https://mlsanity.com/paper/2409.12466/) (2024)
- [Ground-Truth, Whose Truth? -- Examining the Challenges with Annotating Toxic Text Datasets](https://mlsanity.com/paper/2112.03529/) (2021)
- [SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing](https://mlsanity.com/paper/2601.09385/) (2026)
- [Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Data](https://mlsanity.com/paper/2410.02056/) (2024)
- [IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling](https://mlsanity.com/paper/2506.00736/) (2025)
- [Describing emotions with acoustic property prompts for speech emotion recognition](https://mlsanity.com/paper/2211.07737/) (2022)
- [Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction](https://mlsanity.com/paper/2507.09834/) (2025)
- [SAO-Instruct: Free-form Audio Editing using Natural Language Instructions](https://mlsanity.com/paper/2510.22795/) (2025)
- [Unsupervised Improvement of Audio-Text Cross-Modal Representations](https://mlsanity.com/paper/2305.01864/) (2023)
- [Prompting Audios Using Acoustic Properties For Emotion Representation](https://mlsanity.com/paper/2310.02298/) (2023)
- [Learning Audio Concepts from Counterfactual Natural Language](https://mlsanity.com/paper/2401.04935/) (2024)
- [Listenable Maps for Zero-Shot Audio Classifiers](https://mlsanity.com/paper/2405.17615/) (2024)
- [From Text to Talk: Audio-Language Model Needs Non-Autoregressive Joint Training](https://mlsanity.com/paper/2509.20072/) (2025)
- [On Class Separability Pitfalls In Audio-Text Contrastive Zero-Shot Learning](https://mlsanity.com/paper/2408.13068/) (2024)
- [Keep what you need : extracting efficient subnetworks from large audio representation models](https://mlsanity.com/paper/2502.12925/) (2025)
- [Eureka-Audio: Triggering Audio Intelligence in Compact Language Models](https://mlsanity.com/paper/2602.13954/) (2026)
- [AudioSetMix: Enhancing Audio-Language Datasets with LLM-Assisted Augmentations](https://mlsanity.com/paper/2405.11093/) (2024)
- [Representation-Regularized Convolutional Audio Transformer for Audio Understanding](https://mlsanity.com/paper/2601.21612/) (2026)
- [Heterogeneity-Aware Dataset Scheduling for Efficient Audio Large Language Model Training](https://mlsanity.com/paper/2605.19101/) (2026)

## Abstract (arXiv)

Crowdsourcing has become a common approach for annotating large amounts of data. It has the advantage of harnessing a large workforce to produce large amounts of data in a short time, but comes with the disadvantage of employing non-expert annotators with different backgrounds. This raises the problem of data reliability, in addition to the general question of how to combine the opinions of multiple annotators in order to estimate the ground truth. This paper presents a study of the annotations and annotators' reliability for audio tagging. We adapt the use of Krippendorf's alpha and multi-annotator competence estimation (MACE) for a multi-labeled data scenario, and present how MACE can be used to estimate a candidate ground truth based on annotations from non-expert users with different levels of expertise and competence.