# Self-Attention Channel Combinator Frontend for End-to-End Multichannel Far-field Speech Recognition

arXiv: https://arxiv.org/abs/2109.04783 · HTML: https://mlsanity.com/paper/2109.04783/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 25 references that are in the map.
- First posted on arXiv 10 Sep 2021.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Towards better decoding and language model integration in sequence to sequence models](https://mlsanity.com/paper/1612.02695/) (2016)
- [ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context](https://mlsanity.com/paper/2005.03191/) (2020)
- [Advances in Joint CTC-Attention based End-to-End Speech Recognition with a Deep CNN Encoder and RNN-LM](https://mlsanity.com/paper/1706.02737/) (2017)
- [Far-Field Automatic Speech Recognition](https://mlsanity.com/paper/2009.09395/) (2020)
- [Channel-Attention Dense U-Net for Multichannel Speech Enhancement](https://mlsanity.com/paper/2001.11542/) (2020)
- [Multichannel End-to-end Speech Recognition](https://mlsanity.com/paper/1703.04783/) (2017)
- [End-to-End Multi-Channel Transformer for Speech Recognition](https://mlsanity.com/paper/2102.03951/) (2021)
- [End-to-End Dereverberation, Beamforming, and Speech Recognition with Improved Numerical Stability and Advanced Frontend](https://mlsanity.com/paper/2102.11525/) (2021)
- [End-to-End Far-Field Speech Recognition with Unified Dereverberation and Beamforming](https://mlsanity.com/paper/2005.10479/) (2020)
- [Frequency Domain Multi-channel Acoustic Modeling for Distant Speech Recognition](https://mlsanity.com/paper/1903.05299/) (2019)
- [Coherent-to-Diffuse Power Ratio Estimation for Dereverberation](https://mlsanity.com/paper/1502.03784/) (2015)
- [Robust Multi-channel Speech Recognition using Frequency Aligned Network](https://mlsanity.com/paper/2002.02520/) (2020)

## Built on by (most cited first, 4 of 4)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Joint speech and overlap detection: a benchmark over multiple audio setup and speech domains](https://mlsanity.com/paper/2307.13012/) (2023)
- [ASoBO: Attentive Beamformer Selection for Distant Speaker Diarization in Meetings](https://mlsanity.com/paper/2406.03251/) (2024)
- [Spatial Processing Front-End For Distant ASR Exploiting Self-Attention Channel Combinator](https://mlsanity.com/paper/2203.13919/) (2022)

## Abstract (arXiv)

When a sufficiently large far-field training data is presented, jointly optimizing a multichannel frontend and an end-to-end (E2E) Automatic Speech Recognition (ASR) backend shows promising results. Recent literature has shown traditional beamformer designs, such as MVDR (Minimum Variance Distortionless Response) or fixed beamformers can be successfully integrated as the frontend into an E2E ASR system with learnable parameters. In this work, we propose the self-attention channel combinator (SACC) ASR frontend, which leverages the self-attention mechanism to combine multichannel audio signals in the magnitude spectral domain. Experiments conducted on a multichannel playback test data shows that the SACC achieved a 9.3% WERR compared to a state-of-the-art fixed beamformer-based frontend, both jointly optimized with a ContextNet-based ASR backend. We also demonstrate the connection between the SACC and the traditional beamformers, and analyze the intermediate outputs of the SACC.