# Challenges and Insights: Exploring 3D Spatial Features and Complex Networks on the MISP Dataset

arXiv: https://arxiv.org/abs/2310.03901 · HTML: https://mlsanity.com/paper/2310.03901/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Speaker recognition and diarization, one of 11 directions in Speech and audio language models.
- Builds on 6 of its 11 references that are in the map.
- First posted on arXiv 5 Oct 2023.

## Placed in

- Speech and audio language models / [Speaker recognition and diarization](https://mlsanity.com/speech/speaker-recognition-and-diarization/)

## Builds on (most cited first, 6 of 6)

- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement](https://mlsanity.com/paper/2008.00264/) (2020)
- [Pruned RNN-T for fast, memory-efficient ASR training](https://mlsanity.com/paper/2206.13236/) (2022)
- [Multi-modal Multi-channel Target Speech Separation](https://mlsanity.com/paper/2003.07032/) (2020)
- [Multi-Channel Multi-Speaker ASR Using 3D Spatial Feature](https://mlsanity.com/paper/2111.11023/) (2021)
- [Complex Neural Spatial Filter: Enhancing Multi-channel Target Speech Separation in Complex Domain](https://mlsanity.com/paper/2104.12359/) (2021)

## Abstract (arXiv)

Multi-channel multi-talker speech recognition presents formidable challenges in the realm of speech processing, marked by issues such as background noise, reverberation, and overlapping speech. Overcoming these complexities requires leveraging contextual cues to separate target speech from a cacophonous mix, enabling accurate recognition. Among these cues, the 3D spatial feature has emerged as a cutting-edge solution, particularly when equipped with spatial information about the target speaker. Its exceptional ability to discern the target speaker within mixed audio, often rendering intermediate processing redundant, paves the way for the direct training of "All-in-one" ASR models. These models have demonstrated commendable performance on both simulated and real-world data. In this paper, we extend this approach to the MISP dataset to further validate its efficacy. We delve into the challenges encountered and insights gained when applying 3D spatial features to MISP, while also exploring preliminary experiments involving the replacement of these features with more complex input and models.