# Speaker conditioning of acoustic models using affine transformation for multi-speaker speech recognition

arXiv: https://arxiv.org/abs/2111.00320 · HTML: https://mlsanity.com/paper/2111.00320/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 34 references that are in the map.
- First posted on arXiv 30 Oct 2021.

## Builds on (most cited first, 17 of 17)

- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings](https://mlsanity.com/paper/2004.09249/) (2020)
- [Intermediate Loss Regularization for CTC-based Speech Recognition](https://mlsanity.com/paper/2102.03216/) (2021)
- [The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselines](https://mlsanity.com/paper/1803.10609/) (2018)
- [Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predict](https://mlsanity.com/paper/2005.08700/) (2020)
- [End-to-End Multi-speaker Speech Recognition with Transformer](https://mlsanity.com/paper/2002.03921/) (2020)
- [Align-Refine: Non-Autoregressive Speech Recognition via Iterative Realignment](https://mlsanity.com/paper/2010.14233/) (2020)
- [A Purely End-to-end System for Multi-speaker Speech Recognition](https://mlsanity.com/paper/1805.05826/) (2018)
- [Streaming end-to-end multi-talker speech recognition](https://mlsanity.com/paper/2011.13148/) (2020)
- [Optimal Completion Distillation for Sequence Learning](https://mlsanity.com/paper/1810.01398/) (2018)
- [Probabilistic Permutation Invariant Training for Speech Separation](https://mlsanity.com/paper/1908.01768/) (2019)
- [End-to-End Multi-Speaker Speech Recognition using Speaker Embeddings and Transfer Learning](https://mlsanity.com/paper/1908.04737/) (2019)
- [Progressive Joint Modeling in Unsupervised Single-channel Overlapped Speech Recognition](https://mlsanity.com/paper/1707.07048/) (2017)
- [Frame-based overlapping speech detection using Convolutional Neural Networks](https://mlsanity.com/paper/2001.09937/) (2020)
- [End-to-end Anchored Speech Recognition](https://mlsanity.com/paper/1902.02383/) (2019)
- [Do End-to-End Speech Recognition Models Care About Context?](https://mlsanity.com/paper/2102.09928/) (2021)

## Built on by (most cited first, 2 of 2)

- [Single-channel speech separation using Soft-minimum Permutation Invariant Training](https://mlsanity.com/paper/2111.08635/) (2021)
- [Closing the Gap between Single-User and Multi-User VoiceFilter-Lite](https://mlsanity.com/paper/2202.12169/) (2022)

## Abstract (arXiv)

This study addresses the problem of single-channel Automatic Speech Recognition of a target speaker within an overlap speech scenario. In the proposed method, the hidden representations in the acoustic model are modulated by speaker auxiliary information to recognize only the desired speaker. Affine transformation layers are inserted into the acoustic model network to integrate speaker information with the acoustic features. The speaker conditioning process allows the acoustic model to perform computation in the context of target-speaker auxiliary information. The proposed speaker conditioning method is a general approach and can be applied to any acoustic model architecture. Here, we employ speaker conditioning on a ResNet acoustic model. Experiments on the WSJ corpus show that the proposed speaker conditioning method is an effective solution to fuse speaker auxiliary information with acoustic features for multi-speaker speech recognition, achieving +9% and +20% relative WER reduction for clean and overlap speech scenarios, respectively, compared to the original ResNet acoustic model baseline.