# Lightweight and Robust Multi-Channel End-to-End Speech Recognition with Spherical Harmonic Transform

arXiv: https://arxiv.org/abs/2506.11630 · HTML: https://mlsanity.com/paper/2506.11630/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Speaker recognition and diarization, one of 11 directions in Speech and audio language models.
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 13 Jun 2025.

## Placed in

- Speech and audio language models / [Speaker recognition and diarization](https://mlsanity.com/speech/speaker-recognition-and-diarization/)

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Coordinate Attention for Efficient Mobile Network Design](https://mlsanity.com/paper/2103.02907/) (2021)
- [M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challenge](https://mlsanity.com/paper/2110.07393/) (2021)
- [AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation, Recognition and Speaker Diarization in Conference Scenario](https://mlsanity.com/paper/2104.03603/) (2021)
- [TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separation](https://mlsanity.com/paper/2211.12433/) (2022)
- [MIMO-SPEECH: End-to-End Multi-Channel Multi-Speaker Speech Recognition](https://mlsanity.com/paper/1910.06522/) (2019)
- [Multi-microphone Complex Spectral Mapping for Utterance-wise and Continuous Speech Separation](https://mlsanity.com/paper/2010.01703/) (2020)
- [Embedding and Beamforming: All-neural Causal Beamformer for Multichannel Speech Enhancement](https://mlsanity.com/paper/2109.00265/) (2021)
- [Multi-Channel Transformer Transducer for Speech Recognition](https://mlsanity.com/paper/2108.12953/) (2021)
- [CUSIDE: Chunking, Simulating Future Context and Decoding for Streaming ASR](https://mlsanity.com/paper/2203.16758/) (2022)
- [CAT: A CTC-CRF based ASR Toolkit Bridging the Hybrid and the End-to-end Approaches towards Data Efficiency and Low Latency](https://mlsanity.com/paper/2005.13326/) (2020)
- [MFCCA:Multi-Frame Cross-Channel attention for multi-speaker ASR in Multi-party meeting scenario](https://mlsanity.com/paper/2210.05265/) (2022)
- [Self-Attention Channel Combinator Frontend for End-to-End Multichannel Far-field Speech Recognition](https://mlsanity.com/paper/2109.04783/) (2021)
- [Exploiting Single-Channel Speech for Multi-Channel End-to-End Speech Recognition: A Comparative Study](https://mlsanity.com/paper/2203.16757/) (2022)
- [Efficient Multi-Channel Speech Enhancement with Spherical Harmonics Injection for Directional Encoding](https://mlsanity.com/paper/2309.10832/) (2023)
- [Spatial Processing Front-End For Distant ASR Exploiting Self-Attention Channel Combinator](https://mlsanity.com/paper/2203.13919/) (2022)
- [CUSIDE-array: A Streaming Multi-Channel End-to-End Speech Recognition System with Realistic Evaluations](https://mlsanity.com/paper/2407.09807/) (2024)

## Abstract (arXiv)

This paper presents SHTNet, a lightweight spherical harmonic transform (SHT) based framework, which is designed to address cross-array generalization challenges in multi-channel automatic speech recognition (ASR) through three key innovations. First, SHT based spatial sound field decomposition converts microphone signals into geometry-invariant spherical harmonic coefficients, isolating signal processing from array geometry. Second, the Spatio-Spectral Attention Fusion Network (SSAFN) combines coordinate-aware spatial modeling, refined self-attention channel combinator, and spectral noise suppression without conventional beamforming. Third, Rand-SHT training enhances robustness through random channel selection and array geometry reconstruction. The system achieves 39.26% average CER across heterogeneous arrays (e.g., circular, square, and binaural) on datasets including Aishell-4, Alimeeting, and XMOS, with 97.1% fewer computations than conventional neural beamformers.