# Multi-View Spectrogram Transformer for Respiratory Sound Classification

arXiv: https://arxiv.org/abs/2311.09655 · HTML: https://mlsanity.com/paper/2311.09655/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 26 references that are in the map.
- First posted on arXiv 16 Nov 2023.

## Builds on (most cited first, 12 of 12)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [SSAST: Self-Supervised Audio Spectrogram Transformer](https://mlsanity.com/paper/2110.09784/) (2021)
- [VLT: Vision-Language Transformer and Query Generation for Referring Segmentation](https://mlsanity.com/paper/2210.15871/) (2022)
- [Deep Learning for Audio Signal Processing](https://mlsanity.com/paper/1905.00078/) (2019)
- [RespireNet: A Deep Neural Network for Accurately Detecting Abnormal Lung Sounds in Limited Data Setting](https://mlsanity.com/paper/2011.00196/) (2020)
- [Patch-Mix Contrastive Learning with Audio Spectrogram Transformer on Respiratory Sound Classification](https://mlsanity.com/paper/2305.14032/) (2023)
- [Pretraining Respiratory Sound Representations using Metadata and Contrastive Learning](https://mlsanity.com/paper/2210.16192/) (2022)
- [Lung Sound Classification Using Co-tuning and Stochastic Normalization](https://mlsanity.com/paper/2108.01991/) (2021)
- [Attention-based Dual-stream Vision Transformer for Radar Gait Recognition](https://mlsanity.com/paper/2111.12290/) (2021)

## Built on by (most cited first, 3 of 3)

- [Improving Deep Learning-based Respiratory Sound Analysis with Frequency Selection and Attention Mechanism](https://mlsanity.com/paper/2507.20052/) (2025)
- [Geometry-Aware Optimization for Respiratory Sound Classification: Enhancing Sensitivity with SAM-Optimized Audio Spectrogram Transformers](https://mlsanity.com/paper/2512.22564/) (2025)
- [Resp-Agent: An Agent-Based System for Multimodal Respiratory Sound Generation and Disease Diagnosis](https://mlsanity.com/paper/2602.15909/) (2026)

## Abstract (arXiv)

Deep neural networks have been applied to audio spectrograms for respiratory sound classification. Existing models often treat the spectrogram as a synthetic image while overlooking its physical characteristics. In this paper, a Multi-View Spectrogram Transformer (MVST) is proposed to embed different views of time-frequency characteristics into the vision transformer. Specifically, the proposed MVST splits the mel-spectrogram into different sized patches, representing the multi-view acoustic elements of a respiratory sound. These patches and positional embeddings are then fed into transformer encoders to extract the attentional information among patches through a self-attention mechanism. Finally, a gated fusion scheme is designed to automatically weigh the multi-view features to highlight the best one in a specific scenario. Experimental results on the ICBHI dataset demonstrate that the proposed MVST significantly outperforms state-of-the-art methods for classifying respiratory sounds.