# Two-pass Decoding and Cross-adaptation Based System Combination of End-to-end Conformer and Hybrid TDNN ASR Systems

arXiv: https://arxiv.org/abs/2206.11596 · HTML: https://mlsanity.com/paper/2206.11596/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 53 references that are in the map.
- First posted on arXiv 23 Jun 2022.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [A Comparative Study on Transformer vs RNN in Speech Applications](https://mlsanity.com/paper/1909.06317/) (2019)
- [Recent Developments on ESPnet Toolkit Boosted by Conformer](https://mlsanity.com/paper/2010.13956/) (2020)
- [Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1402.1128/) (2014)
- [Two-Pass End-to-End Speech Recognition](https://mlsanity.com/paper/1908.10992/) (2019)
- [Learning Hidden Unit Contributions for Unsupervised Acoustic Model Adaptation](https://mlsanity.com/paper/1601.02828/) (2016)
- [Single headed attention based sequence-to-sequence model for state-of-the-art results on Switchboard](https://mlsanity.com/paper/2001.07263/) (2020)
- [On the limit of English conversational speech recognition](https://mlsanity.com/paper/2105.00982/) (2021)
- [Less Is More: Improved RNN-T Decoding Using Limited Label Context and Path Merging](https://mlsanity.com/paper/2012.06749/) (2020)
- [An investigation of phone-based subword units for end-to-end speech recognition](https://mlsanity.com/paper/2004.04290/) (2020)
- [Cumulative Adaptation for BLSTM Acoustic Models](https://mlsanity.com/paper/1906.06207/) (2019)
- [Bayesian Learning for Deep Neural Network Adaptation](https://mlsanity.com/paper/2012.07460/) (2020)
- [Integrating Source-channel and Attention-based Sequence-to-sequence Models for Speech Recognition](https://mlsanity.com/paper/1909.06614/) (2019)

## Built on by (most cited first, 4 of 4)

- [Exploring Self-supervised Pre-trained ASR Models For Dysarthric and Elderly Speech Recognition](https://mlsanity.com/paper/2302.14564/) (2023)
- [Exploring linguistic feature and model combination for speech recognition based automatic AD detection](https://mlsanity.com/paper/2206.13758/) (2022)
- [Self-supervised ASR Models and Features For Dysarthric and Elderly Speech Recognition](https://mlsanity.com/paper/2407.13782/) (2024)
- [Use of Speech Impairment Severity for Dysarthric Speech Recognition](https://mlsanity.com/paper/2305.10659/) (2023)

## Abstract (arXiv)

Fundamental modelling differences between hybrid and end-to-end (E2E) automatic speech recognition (ASR) systems create large diversity and complementarity among them. This paper investigates multi-pass rescoring and cross adaptation based system combination approaches for hybrid TDNN and Conformer E2E ASR systems. In multi-pass rescoring, state-of-the-art hybrid LF-MMI trained CNN-TDNN system featuring speed perturbation, SpecAugment and Bayesian learning hidden unit contributions (LHUC) speaker adaptation was used to produce initial N-best outputs before being rescored by the speaker adapted Conformer system using a 2-way cross system score interpolation. In cross adaptation, the hybrid CNN-TDNN system was adapted to the 1-best output of the Conformer system or vice versa. Experiments on the 300-hour Switchboard corpus suggest that the combined systems derived using either of the two system combination approaches outperformed the individual systems. The best combined system obtained using multi-pass rescoring produced statistically significant word error rate (WER) reductions of 2.5% to 3.9% absolute (22.5% to 28.9% relative) over the stand alone Conformer system on the NIST Hub5'00, Rt03 and Rt02 evaluation data.