# RIR-SF: Room Impulse Response Based Spatial Feature for Target Speech Recognition in Multi-Channel Multi-Speaker Scenarios

arXiv: https://arxiv.org/abs/2311.00146 · HTML: https://mlsanity.com/paper/2311.00146/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 31 Oct 2023.

## Builds on (most cited first, 17 of 17)

- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017)
- [Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2010.10504/) (2020)
- [Pyroomacoustics: A Python package for audio room simulations and array processing algorithms](https://mlsanity.com/paper/1710.04196/) (2017)
- [Pruned RNN-T for fast, memory-efficient ASR training](https://mlsanity.com/paper/2206.13236/) (2022)
- [Far-Field Automatic Speech Recognition](https://mlsanity.com/paper/2009.09395/) (2020)
- [Espresso: A Fast End-to-end Neural Speech Recognition Toolkit](https://mlsanity.com/paper/1909.08723/) (2019)
- [Time Domain Audio Visual Speech Separation](https://mlsanity.com/paper/1904.03760/) (2019)
- [Multi-modal Multi-channel Target Speech Separation](https://mlsanity.com/paper/2003.07032/) (2020)
- [ADL-MVDR: All deep learning MVDR beamformer for target speech separation](https://mlsanity.com/paper/2008.06994/) (2020)
- [PyChain: A Fully Parallelized PyTorch Implementation of LF-MMI for End-to-End ASR](https://mlsanity.com/paper/2005.09824/) (2020)
- [Audio-visual Multi-channel Integration and Recognition of Overlapped Speech](https://mlsanity.com/paper/2011.07755/) (2020)
- [Multi-Channel Multi-Speaker ASR Using 3D Spatial Feature](https://mlsanity.com/paper/2111.11023/) (2021)
- [End-to-End Integration of Speech Recognition, Dereverberation, Beamforming, and Self-Supervised Learning Representation](https://mlsanity.com/paper/2210.10742/) (2022)
- [Towards Unified All-Neural Beamforming for Time and Frequency Domain Speech Separation](https://mlsanity.com/paper/2212.08348/) (2022)
- [Audio-visual End-to-end Multi-channel Speech Separation, Dereverberation and Recognition](https://mlsanity.com/paper/2307.02909/) (2023)
- [Multichannel Speech Separation and Enhancement Using the Convolutive Transfer Function](https://mlsanity.com/paper/1711.07911/) (2017)

## Abstract (arXiv)

Automatic speech recognition (ASR) on multi-talker recordings is challenging. Current methods using 3D spatial data from multi-channel audio and visual cues focus mainly on direct waves from the target speaker, overlooking reflection wave impacts, which hinders performance in reverberant environments. Our research introduces RIR-SF, a novel spatial feature based on room impulse response (RIR) that leverages the speaker's position, room acoustics, and reflection dynamics. RIR-SF significantly outperforms traditional 3D spatial features, showing superior theoretical and empirical performance. We also propose an optimized all-neural multi-channel ASR framework for RIR-SF, achieving a relative 21.3% reduction in CER for target speaker ASR in multi-channel settings. RIR-SF enhances recognition accuracy and demonstrates robustness in high-reverberation scenarios, overcoming the limitations of previous methods.