mlsanityData to 7 Oct 2026

What is the ground truth? Reliability of multi-annotator data for audio tagging

arXiv 2104.04214 ↗First posted 9 Apr 2021eess.AScited by 29

Not on the map. Why?

Abstract (arXiv)

Crowdsourcing has become a common approach for annotating large amounts of data. It has the advantage of harnessing a large workforce to produce large amounts of data in a short time, but comes with the disadvantage of employing non-expert annotators with different backgrounds. This raises the problem of data reliability, in addition to the general question of how to combine the opinions of multiple annotators in order to estimate the ground truth. This paper presents a study of the annotations and annotators' reliability for audio tagging. We adapt the use of Krippendorf's alpha and multi-annotator competence estimation (MACE) for a multi-labeled data scenario, and present how MACE can be used to estimate a candidate ground truth based on annotations from non-expert users with different levels of expertise and competence.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Apr 2021.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 2; 2023: 4; 2024: 8; 2025: 9; 2026: 5.

29 papers.

Sort:

By area:

  1. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  2. 2025Kimi-Audio Technical Reportcited by 210
  3. 2023Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generationcited by 90
  4. 2025JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronizationcited by 84
  5. 2024Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generationcited by 54
  6. 2022Audio Retrieval with WavText5K and CLAP Trainingcited by 35
  7. 2026JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generationcited by 22
  8. 2024AudioLCM: Text-to-Audio Generation with Latent Consistency Modelscited by 21
  9. 2025The iNaturalist Sounds Datasetcited by 17
  10. 2024AudioEditor: A Training-Free Diffusion-Based Audio Editing Frameworkcited by 13
  11. 2025Object-AVEdit: An Object-level Audio-Visual Editing Modelcited by 12
  12. 2021Ground-Truth, Whose Truth? -- Examining the Challenges with Annotating Toxic Text Datasetscited by 11
17 more
  1. 2024Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Datacited by 7
  2. 2026SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processingcited by 7
  3. 2025IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modelingcited by 6
  4. 2022Describing emotions with acoustic property prompts for speech emotion recognitioncited by 5
  5. 2025Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Predictioncited by 5
  6. 2023Unsupervised Improvement of Audio-Text Cross-Modal Representationscited by 3
  7. 2023Prompting Audios Using Acoustic Properties For Emotion Representationcited by 3
  8. 2024Learning Audio Concepts from Counterfactual Natural Languagecited by 3
  9. 2024Listenable Maps for Zero-Shot Audio Classifierscited by 3
  10. 2025From Text to Talk: Audio-Language Model Needs Non-Autoregressive Joint Trainingcited by 3
  11. 2025SAO-Instruct: Free-form Audio Editing using Natural Language Instructionscited by 3
  12. 2024On Class Separability Pitfalls In Audio-Text Contrastive Zero-Shot Learningcited by 1
  13. 2025Keep what you need : extracting efficient subnetworks from large audio representation modelscited by 1
  14. 2024AudioSetMix: Enhancing Audio-Language Datasets with LLM-Assisted Augmentationscited by 0
  15. 2026Representation-Regularized Convolutional Audio Transformer for Audio Understandingcited by 0
  16. 2026Eureka-Audio: Triggering Audio Intelligence in Compact Language Modelscited by 0
  17. 2026Heterogeneity-Aware Dataset Scheduling for Efficient Audio Large Language Model Trainingcited by 0