mlsanityData to 14 Oct 2026

I can listen but cannot read: An evaluation of two-tower multimodal systems for instrument recognition

arXiv 2407.18058 ↗First posted 25 Jul 2024cs.SD cs.CL cs.IRcited by 4

Not on the map. Why?

Abstract (arXiv)

Music two-tower multimodal systems integrate audio and text modalities into a joint audio-text space, enabling direct comparison between songs and their corresponding labels. These systems enable new approaches for classification and retrieval, leveraging both modalities. Despite the promising results they have shown for zero-shot classification and retrieval tasks, closer inspection of the embeddings is needed. This paper evaluates the inherent zero-shot properties of joint audio-text spaces for the case-study of instrument recognition. We present an evaluation and analysis of two-tower systems for zero-shot instrument recognition and a detailed analysis of the properties of the pre-joint and joint embeddings spaces. Our findings suggest that audio encoders alone demonstrate good quality, while challenges remain within the text encoder or joint space projection. Specifically, two-tower systems exhibit sensitivity towards specific words, favoring generic prompts over musically informed ones. Despite the large size of textual encoders, they do not yet leverage additional textual context or infer instruments accurately from their descriptions. Lastly, a novel approach for quantifying the semantic meaningfulness of the textual space leveraging an instrument ontology is proposed. This method reveals deficiencies in the systems' understanding of instruments and provides evidence of the need for fine-tuning text encoders on musical data.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 37 of its 53 references that are in the map. First posted on arXiv 25 Jul 2024.

Builds on

37 of 53 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  8. 2019Contrastive Multiview Codingcited by 1,301
  9. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  10. 2021AST: Audio Spectrogram Transformercited by 643
  11. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  12. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 574
25 more
  1. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 267
  2. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 176
  3. 2021Active Learning by Acquiring Contrastive Examplescited by 156
  4. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 150
  5. 2022MuLan: A Joint Embedding of Music Audio and Natural Languagecited by 87
  6. 2023LP-MusicCaps: LLM-Based Pseudo Music Captioningcited by 83
  7. 2021Towards Learning Universal Audio Representationscited by 51
  8. 2022Improving Zero-shot Generalization and Robustness of Multi-modal Modelscited by 45
  9. 2023LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collectionscited by 43
  10. 2020Evaluation of CNN-based Automatic Music Tagging Modelscited by 41
  11. 2022Contrastive Audio-Language Learning for Musiccited by 33
  12. 2020Zero-Shot Audio Classification via Semantic Embeddingscited by 21
  13. 2022Toward Universal Text-to-Music Retrievalcited by 18
  14. 2016A Powerful Generative Model Using Random Weights for the Deep Image Representationcited by 14
  15. 2019Zero-shot Learning for Audio-based Music Classification and Taggingcited by 13
  16. 2022Improving Multimodal Speech Recognition by Data Augmentation and Speech Representationscited by 13
  17. 2019Zero-Shot Audio Classification Based on Class Label Embeddingscited by 9
  18. 2021Music Classification: Beyond Supervised Learning, Towards Real-world Applicationscited by 7
  19. 2020Zero-Shot Audio Classification with Factored Linear and Nonlinear Acoustic-Semantic Projectionscited by 6
  20. 2022Modality-Balanced Embedding for Video Retrievalcited by 6
  21. 2020Active Contrastive Learning of Audio-Visual Video Representationscited by 5
  22. 2022Towards Proper Contrastive Self-supervised Learning Strategies For Music Audio Representationcited by 5
  23. 2023Joint Music and Language Attention Models for Zero-shot Music Taggingcited by 5
  24. 2018Audio Based Disambiguation Of Music Genre Tagscited by 4
  25. 2022Zero-Shot Audio Classification using Image Embeddingscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2024Multimodal Alignment and Fusion: A Surveycited by 74
  2. 2025MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantizationcited by 68
  3. 2026Towards Effective Negation Modeling in Joint Audio-Text Models for Musiccited by 1
  4. 2025Capturing Fine-Grained Alignments Improves 3D Affordance Detectioncited by 0