# Listen, Read, and Identify: Multimodal Singing Language Identification of Music

arXiv: https://arxiv.org/abs/2103.01893 · HTML: https://mlsanity.com/paper/2103.01893/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 24 references that are in the map.
- First posted on arXiv 2 Mar 2021.

## Builds on (most cited first, 7 of 7)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Evaluation of CNN-based Automatic Music Tagging Models](https://mlsanity.com/paper/2006.00751/) (2020)
- [Automatic tagging using deep convolutional neural networks](https://mlsanity.com/paper/1606.00298/) (2016)
- [Kapre: On-GPU Audio Preprocessing Layers for a Quick Implementation of Deep Neural Network Models with Keras](https://mlsanity.com/paper/1706.05781/) (2017)
- [MaD TwinNet: Masker-Denoiser Architecture with Twin Networks for Monaural Sound Source Separation](https://mlsanity.com/paper/1802.00300/) (2018)

## Built on by (most cited first, 2 of 2)

- [More than words: Advancements and challenges in speech recognition for singing](https://mlsanity.com/paper/2403.09298/) (2024)
- [Multimodal Modeling For Spoken Language Identification](https://mlsanity.com/paper/2309.10567/) (2023)

## Abstract (arXiv)

We propose a multimodal singing language classification model that uses both audio content and textual metadata. LRID-Net, the proposed model, takes an audio signal and a language probability vector estimated from the metadata and outputs the probabilities of the target languages. Optionally, LRID-Net is facilitated with modality dropouts to handle a missing modality. In the experiment, we trained several LRID-Nets with varying modality dropout configuration and tested them with various combinations of input modalities. The experiment results demonstrate that using multimodal input improves performance. The results also suggest that adopting modality dropout does not degrade the performance of the model when there are full modality inputs while enabling the model to handle missing modality cases to some extent.