# Mask-based Neural Beamforming for Moving Speakers with Self-Attention-based Tracking

arXiv: https://arxiv.org/abs/2205.03568 · HTML: https://mlsanity.com/paper/2205.03568/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 May 2022.

## Built on by (most cited first, 4 of 4)

- [Autoregressive Guidance of Deep Spatially Selective Filters using Bayesian Tracking for Efficient Extraction of Moving Speakers](https://mlsanity.com/paper/2603.23723/) (2026)
- [Attention-Driven Multichannel Speech Enhancement in Moving Sound Source Scenarios](https://mlsanity.com/paper/2312.10756/) (2023)
- [Visual-Informed Speech Enhancement Using Attention-Based Beamforming](https://mlsanity.com/paper/2603.05270/) (2026)
- [Weakly Guided and Autoregressive Beamformer Parameterization for Generalizable Moving Speaker Extraction in Higher-Order Ambisonics](https://mlsanity.com/paper/2607.04471/) (2026)

## Abstract (arXiv)

Beamforming is a powerful tool designed to enhance speech signals from the direction of a target source. Computing the beamforming filter requires estimating spatial covariance matrices (SCMs) of the source and noise signals. Time-frequency masks are often used to compute these SCMs. Most studies of mask-based beamforming have assumed that the sources do not move. However, sources often move in practice, which causes performance degradation. In this paper, we address the problem of mask-based beamforming for moving sources. We first review classical approaches to tracking a moving source, which perform online or blockwise computation of the SCMs. We show that these approaches can be interpreted as computing a sum of instantaneous SCMs weighted by attention weights. These weights indicate which time frames of the signal to consider in the SCM computation. Online or blockwise computation assumes a heuristic and deterministic way of computing these attention weights that, although simple, may not result in optimal performance. We thus introduce a learning-based framework that computes optimal attention weights for beamforming. We achieve this using a neural network implemented with self-attention layers. We show experimentally that our proposed framework can greatly improve beamforming performance in moving source situations while maintaining high performance in non-moving situations, thus enabling the development of mask-based beamformers robust to source movements.