# An Explainable Proxy Model for Multiabel Audio Segmentation

arXiv: https://arxiv.org/abs/2401.08268 · HTML: https://mlsanity.com/paper/2401.08268/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 31 references that are in the map.
- First posted on arXiv 16 Jan 2024.

## Builds on (most cited first, 13 of 13)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [End-to-end speaker segmentation for overlap-aware resegmentation](https://mlsanity.com/paper/2104.04045/) (2021)
- [Visualizing Deep Networks by Optimizing with Integrated Gradients](https://mlsanity.com/paper/1905.00954/) (2019)
- [The Third DIHARD Diarization Challenge](https://mlsanity.com/paper/2012.01477/) (2020)
- [Overlap-aware diarization: resegmentation using neural end-to-end overlapped speech detection](https://mlsanity.com/paper/1910.11646/) (2019)
- [Explaining deep learning models for spoofing and deepfake detection with SHapley Additive exPlanations](https://mlsanity.com/paper/2110.03309/) (2021)
- [End-to-end Domain-Adversarial Voice Activity Detection](https://mlsanity.com/paper/1910.10655/) (2019)
- [Three-class Overlapped Speech Detection using a Convolutional Recurrent Neural Network](https://mlsanity.com/paper/2104.02878/) (2021)
- [Overlapped speech and gender detection with WavLM pre-trained features](https://mlsanity.com/paper/2209.04167/) (2022)
- [Joint speech and overlap detection: a benchmark over multiple audio setup and speech domains](https://mlsanity.com/paper/2307.13012/) (2023)

## Built on by (most cited first, 1 of 1)

- [Explainable by-design Audio Segmentation through Non-Negative Matrix Factorization and Probing](https://mlsanity.com/paper/2406.13385/) (2024)

## Abstract (arXiv)

Audio signal segmentation is a key task for automatic audio indexing. It consists of detecting the boundaries of class-homogeneous segments in the signal. In many applications, explainable AI is a vital process for transparency of decision-making with machine learning. In this paper, we propose an explainable multilabel segmentation model that solves speech activity (SAD), music (MD), noise (ND), and overlapped speech detection (OSD) simultaneously. This proxy uses the non-negative matrix factorization (NMF) to map the embedding used for the segmentation to the frequency domain. Experiments conducted on two datasets show similar performances as the pre-trained black box model while showing strong explainability features. Specifically, the frequency bins used for the decision can be easily identified at both the segment level (local explanations) and global level (class prototypes).