# Streaming Transformer Transducer Based Speech Recognition Using Non-Causal Convolution

arXiv: https://arxiv.org/abs/2110.05241 · HTML: https://mlsanity.com/paper/2110.05241/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 27 references that are in the map.
- First posted on arXiv 7 Oct 2021.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [A Comparative Study on Transformer vs RNN in Speech Applications](https://mlsanity.com/paper/1909.06317/) (2019)
- [Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View](https://mlsanity.com/paper/1906.02762/) (2019)
- [Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition](https://mlsanity.com/paper/2010.10759/) (2020)
- [Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Dataset](https://mlsanity.com/paper/2010.11395/) (2020)
- [Transformer-based Acoustic Modeling for Hybrid Speech Recognition](https://mlsanity.com/paper/1910.09799/) (2019)
- [Transformer-Transducer: End-to-End Speech Recognition with Self-Attention](https://mlsanity.com/paper/1910.12977/) (2019)
- [FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization](https://mlsanity.com/paper/2010.11148/) (2020)
- [Talking-Heads Attention](https://mlsanity.com/paper/2003.02436/) (2020)
- [Alignment Restricted Streaming Recurrent Neural Network Transducer](https://mlsanity.com/paper/2011.03072/) (2020)
- [Streaming Transformer-based Acoustic Models Using Self-attention with Augmented Memory](https://mlsanity.com/paper/2005.08042/) (2020)
- [Self-Attentional Acoustic Models](https://mlsanity.com/paper/1803.09519/) (2018)
- [Syllable-Based Sequence-to-Sequence Speech Recognition with the Transformer in Mandarin Chinese](https://mlsanity.com/paper/1804.10752/) (2018)
- [Low Latency End-to-End Streaming Speech Recognition with a Scout Network](https://mlsanity.com/paper/2003.10369/) (2020)
- [Faster, Simpler and More Accurate Hybrid ASR Systems Using Wordpieces](https://mlsanity.com/paper/2005.09150/) (2020)
- [Streaming Attention-Based Models with Augmented Memory for End-to-End Speech Recognition](https://mlsanity.com/paper/2011.07120/) (2020)

## Built on by (most cited first, 5 of 5)

- [Speech ReaLLM -- Real-time Streaming Speech Recognition with Multimodal LLMs by Teaching the Flow of Time](https://mlsanity.com/paper/2406.09569/) (2024)
- [Learning ASR pathways: A sparse multilingual ASR model](https://mlsanity.com/paper/2209.05735/) (2022)
- [Deliberation Model for On-Device Spoken Language Understanding](https://mlsanity.com/paper/2204.01893/) (2022)
- [Learning a Dual-Mode Speech Recognition Model via Self-Pruning](https://mlsanity.com/paper/2207.11906/) (2022)
- [Modality Confidence Aware Training for Robust End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2307.12134/) (2023)

## Abstract (arXiv)

This paper improves the streaming transformer transducer for speech recognition by using non-causal convolution. Many works apply the causal convolution to improve streaming transformer ignoring the lookahead context. We propose to use non-causal convolution to process the center block and lookahead context separately. This method leverages the lookahead context in convolution and maintains similar training and decoding efficiency. Given the similar latency, using the non-causal convolution with lookahead context gives better accuracy than causal convolution, especially for open-domain dictation scenarios. Besides, this paper applies talking-head attention and a novel history context compression scheme to further improve the performance. The talking-head attention improves the multi-head self-attention by transferring information among different heads. The history context compression method introduces more extended history context compactly. On our in-house data, the proposed methods improve a small Emformer baseline with lookahead context by relative WERR 5.1%, 14.5%, 8.4% on open-domain dictation, assistant general scenarios, and assistant calling scenarios, respectively.