# End-to-End Multi-speaker ASR with Independent Vector Analysis

arXiv: https://arxiv.org/abs/2204.00218 · HTML: https://mlsanity.com/paper/2204.00218/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 51 references that are in the map.
- First posted on arXiv 1 Apr 2022.

## Builds on (most cited first, 26 of 26)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning](https://mlsanity.com/paper/1609.06773/) (2016)
- [A Comparative Study on Transformer vs RNN in Speech Applications](https://mlsanity.com/paper/1909.06317/) (2019)
- [Pyroomacoustics: A Python package for audio room simulations and array processing algorithms](https://mlsanity.com/paper/1710.04196/) (2017)
- [The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usage](https://mlsanity.com/paper/2111.09344/) (2021)
- [Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networks](https://mlsanity.com/paper/1703.06284/) (2017)
- [The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselines](https://mlsanity.com/paper/1803.10609/) (2018)
- [A multi-device dataset for urban acoustic scene classification](https://mlsanity.com/paper/1807.09840/) (2018)
- [MIMO-SPEECH: End-to-End Multi-Channel Multi-Speaker Speech Recognition](https://mlsanity.com/paper/1910.06522/) (2019)
- [End-to-End Multi-speaker Speech Recognition with Transformer](https://mlsanity.com/paper/2002.03921/) (2020)
- [Far-Field Automatic Speech Recognition](https://mlsanity.com/paper/2009.09395/) (2020)
- [End-to-end Speech Recognition with Word-based RNN Language Models](https://mlsanity.com/paper/1808.02608/) (2018)
- [Unsupervised training of a deep clustering model for multichannel blind source separation](https://mlsanity.com/paper/1904.01340/) (2019)
- [End-to-End Dereverberation, Beamforming, and Speech Recognition with Improved Numerical Stability and Advanced Frontend](https://mlsanity.com/paper/2102.11525/) (2021)
- [End-to-End Far-Field Speech Recognition with Unified Dereverberation and Beamforming](https://mlsanity.com/paper/2005.10479/) (2020)
- [A unified convolutional beamformer for simultaneous denoising and dereverberation](https://mlsanity.com/paper/1812.08400/) (2018)
- [Surrogate Source Model Learning for Determined Source Separation](https://mlsanity.com/paper/2011.05540/) (2020)
- [Unsupervised Training for Deep Speech Source Separation with Kullback-Leibler Divergence Based Probabilistic Loss Function](https://mlsanity.com/paper/1911.04228/) (2019)
- [Jointly optimal denoising, dereverberation, and source separation](https://mlsanity.com/paper/2005.09843/) (2020)
- [Joint Dereverberation and Separation with Iterative Source Steering](https://mlsanity.com/paper/2102.06322/) (2021)
- [A Comparison and Combination of Unsupervised Blind Source Separation Techniques](https://mlsanity.com/paper/2106.05627/) (2021)
- [Overdetermined independent vector analysis](https://mlsanity.com/paper/2003.02458/) (2020)
- [MM Algorithms for Joint Independent Subspace Analysis with Application to Blind Single and Multi-Source Extraction](https://mlsanity.com/paper/2004.03926/) (2020)
- [Independence-based Joint Dereverberation and Separation with Neural Source Model](https://mlsanity.com/paper/2110.06545/) (2021)

## Built on by (most cited first, 1 of 1)

- [End-to-end Multichannel Speaker-Attributed ASR: Speaker Guided Decoder and Input Feature Analysis](https://mlsanity.com/paper/2310.10106/) (2023)

## Abstract (arXiv)

We develop an end-to-end system for multi-channel, multi-speaker automatic speech recognition. We propose a frontend for joint source separation and dereverberation based on the independent vector analysis (IVA) paradigm. It uses the fast and stable iterative source steering algorithm together with a neural source model. The parameters from the ASR module and the neural source model are optimized jointly from the ASR loss itself. We demonstrate competitive performance with previous systems using neural beamforming frontends. First, we explore the trade-offs when using various number of channels for training and testing. Second, we demonstrate that the proposed IVA frontend performs well on noisy data, even when trained on clean mixtures only. Furthermore, it extends without retraining to the separation of more speakers, which is demonstrated on mixtures of three and four speakers.