# Multi-view Audio and Music Classification

arXiv: https://arxiv.org/abs/2103.02420 · HTML: https://mlsanity.com/paper/2103.02420/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 32 references that are in the map.
- First posted on arXiv 3 Mar 2021.

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [XSleepNet: Multi-View Sequential Model for Automatic Sleep Staging](https://mlsanity.com/paper/2007.05492/) (2020)
- [Learning Problem-agnostic Speech Representations from Multiple Self-supervised Tasks](https://mlsanity.com/paper/1904.03416/) (2019)
- [Very Deep Convolutional Neural Networks for Raw Waveforms](https://mlsanity.com/paper/1610.00087/) (2016)
- [Learning from Between-class Examples for Deep Sound Recognition](https://mlsanity.com/paper/1711.10282/) (2017)
- [Convolutional Recurrent Neural Networks for Music Classification](https://mlsanity.com/paper/1609.04243/) (2016)
- [ESResNet: Environmental Sound Classification Based on Visual Domain Models](https://mlsanity.com/paper/2004.07301/) (2020)
- [Comparison of Time-Frequency Representations for Environmental Sound Classification using Convolutional Neural Networks](https://mlsanity.com/paper/1706.07156/) (2017)
- [SubSpectralNet - Using Sub-Spectrogram based Convolutional Neural Networks for Acoustic Scene Classification](https://mlsanity.com/paper/1810.12642/) (2018)
- [Bottom-up Broadcast Neural Network For Music Genre Classification](https://mlsanity.com/paper/1901.08928/) (2019)
- [Robust Acoustic Scene Classification using a Multi-Spectrogram Encoder-Decoder Framework](https://mlsanity.com/paper/2002.04502/) (2020)
- [Spatio-Temporal Attention Pooling for Audio Scene Classification](https://mlsanity.com/paper/1904.03543/) (2019)
- [Unifying Isolated and Overlapping Audio Event Detection with Multi-Label Multi-Task Convolutional Recurrent Neural Networks](https://mlsanity.com/paper/1811.01092/) (2018)

## Built on by (most cited first, 8 of 8)

- [Wider or Deeper Neural Network Architecture for Acoustic Scene Classification with Mismatched Recording Devices](https://mlsanity.com/paper/2203.12314/) (2022)
- [An Audio-Visual Dataset and Deep Learning Frameworks for Crowded Scene Classification](https://mlsanity.com/paper/2112.09172/) (2021)
- [An Inception-Residual-Based Architecture with Multi-Objective Loss for Detecting Respiratory Anomalies](https://mlsanity.com/paper/2303.04104/) (2023)
- [Low-complexity deep learning frameworks for acoustic scene classification](https://mlsanity.com/paper/2206.06057/) (2022)
- [Robust, General, and Low Complexity Acoustic Scene Classification Systems and An Effective Visualization for Presenting a Sound Scene Context](https://mlsanity.com/paper/2210.08610/) (2022)
- [Multi-modal Multi-view Clustering based on Non-negative Matrix Factorization](https://mlsanity.com/paper/2308.04778/) (2023)
- [Low-complexity deep learning frameworks for acoustic scene classification using teacher-student scheme and multiple spectrograms](https://mlsanity.com/paper/2305.09463/) (2023)
- [Environmental Sound Deepfake Detection Using Deep-Learning Framework](https://mlsanity.com/paper/2604.19652/) (2026)

## Abstract (arXiv)

We propose in this work a multi-view learning approach for audio and music classification. Considering four typical low-level representations (i.e. different views) commonly used for audio and music recognition tasks, the proposed multi-view network consists of four subnetworks, each handling one input types. The learned embedding in the subnetworks are then concatenated to form the multi-view embedding for classification similar to a simple concatenation network. However, apart from the joint classification branch, the network also maintains four classification branches on the single-view embedding of the subnetworks. A novel method is then proposed to keep track of the learning behavior on the classification branches and adapt their weights to proportionally blend their gradients for network training. The weights are adapted in such a way that learning on a branch that is generalizing well will be encouraged whereas learning on a branch that is overfitting will be slowed down. Experiments on three different audio and music classification tasks show that the proposed multi-view network not only outperforms the single-view baselines but also is superior to the multi-view baselines based on concatenation and late fusion.