Enhancing Monotonic Multihead Attention for Streaming ASR
Not on the map. Why?
Abstract (arXiv)
We investigate a monotonic multihead attention (MMA) by extending hard monotonic attention to Transformer-based automatic speech recognition (ASR) for online streaming applications. For streaming inference, all monotonic attention (MA) heads should learn proper alignments because the next token is not generated until all heads detect the corresponding token boundaries. However, we found not all MA heads learn alignments with a na"ive implementation. To encourage every head to learn alignments properly, we propose HeadDrop regularization by masking out a part of heads stochastically during training. Furthermore, we propose to prune redundant heads to improve consensus among heads for boundary detection and prevent delayed token generation caused by such heads. Chunkwise attention on each MA head is extended to the multihead counterpart. Finally, we propose head-synchronous beam search decoding to guarantee stable streaming inference.
Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 May 2020.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 8; 2022: 5; 2023: 2; 2024: 1.
16 papers.
Sort:
- 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
- 2021WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkitcited by 123
- 2021U2++: Unified Two-pass Bidirectional End-to-end Model for Speech Recognitioncited by 28
- 2023Stateful Conformer with Cache-based Inference for Streaming Automatic Speech Recognitioncited by 20
- 2021Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognitioncited by 11
- 2021Streaming End-to-End ASR based on Blockwise Non-Autoregressive Modelscited by 11
- 2022CUSIDE: Chunking, Simulating Future Context and Decoding for Streaming ASRcited by 11
- 2021WNARS: WFST based Non-autoregressive Streaming End-to-End Speech Recognitioncited by 10
- 2023Unified Segment-to-Segment Framework for Simultaneous Sequence Generationcited by 6
- 2021Thank you for Attention: A survey on Attention-based Artificial Neural Networks for Automatic Speech Recognitioncited by 5
- 2024Joint Beam Search Integrating CTC, Attention, and Transducer Decoderscited by 4
- 2022Shifted Chunk Encoder for Transformer Based Streaming End-to-End ASRcited by 2
4 more
- 2022Adaptive Sparse and Monotonic Attention for Transformer-based Automatic Speech Recognitioncited by 2
- 2022SSCFormer: Push the Limit of Chunk-wise Conformer for Streaming ASR Using Sequentially Sampled Chunks and Chunked Causal Convolutioncited by 1
- 2021Mutually-Constrained Monotonic Multihead Attention for Online ASRcited by 0
- 2022FusionFormer: Fusing Operations in Transformer for Efficient Streaming Speech Recognitioncited by 0
- 2024Joint Beam Search Integrating CTC, Attention, and Transducer Decoders5 Jun 2024
- 2023Stateful Conformer with Cache-based Inference for Streaming Automatic Speech Recognition27 Dec 2023
- 2023Unified Segment-to-Segment Framework for Simultaneous Sequence Generation27 Oct 2023
- 2022SSCFormer: Push the Limit of Chunk-wise Conformer for Streaming ASR Using Sequentially Sampled Chunks and Chunked Causal Convolution21 Nov 2022
- 2022FusionFormer: Fusing Operations in Transformer for Efficient Streaming Speech Recognition31 Oct 2022
- 2022Adaptive Sparse and Monotonic Attention for Transformer-based Automatic Speech Recognition30 Sep 2022
- 2022CUSIDE: Chunking, Simulating Future Context and Decoding for Streaming ASR31 Mar 2022
- 2022Shifted Chunk Encoder for Transformer Based Streaming End-to-End ASR29 Mar 2022
- 2021Recent Advances in End-to-End Automatic Speech Recognition2 Nov 2021
- 2021Streaming End-to-End ASR based on Blockwise Non-Autoregressive Models20 Jul 2021
- 2021U2++: Unified Two-pass Bidirectional End-to-end Model for Speech Recognition10 Jun 2021
- 2021WNARS: WFST based Non-autoregressive Streaming End-to-End Speech Recognition8 Apr 2021
4 more
- 2021Mutually-Constrained Monotonic Multihead Attention for Online ASR26 Mar 2021
- 2021Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognition28 Feb 2021
- 2021Thank you for Attention: A survey on Attention-based Artificial Neural Networks for Automatic Speech Recognition14 Feb 2021
- 2021WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit2 Feb 2021
- 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
- 2021WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkitcited by 123
- 2021U2++: Unified Two-pass Bidirectional End-to-end Model for Speech Recognitioncited by 28
- 2023Stateful Conformer with Cache-based Inference for Streaming Automatic Speech Recognitioncited by 20
- 2021Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognitioncited by 11
- 2021Streaming End-to-End ASR based on Blockwise Non-Autoregressive Modelscited by 11
- 2022CUSIDE: Chunking, Simulating Future Context and Decoding for Streaming ASRcited by 11
- 2021WNARS: WFST based Non-autoregressive Streaming End-to-End Speech Recognitioncited by 10
- 2023Unified Segment-to-Segment Framework for Simultaneous Sequence Generationcited by 6
- 2021Thank you for Attention: A survey on Attention-based Artificial Neural Networks for Automatic Speech Recognitioncited by 5
- 2024Joint Beam Search Integrating CTC, Attention, and Transducer Decoderscited by 4
- 2022Shifted Chunk Encoder for Transformer Based Streaming End-to-End ASRcited by 2
12 most cited of 16 All 16 (JSON)
- 2024Joint Beam Search Integrating CTC, Attention, and Transducer Decoders5 Jun 2024
- 2023Stateful Conformer with Cache-based Inference for Streaming Automatic Speech Recognition27 Dec 2023
- 2023Unified Segment-to-Segment Framework for Simultaneous Sequence Generation27 Oct 2023
- 2022SSCFormer: Push the Limit of Chunk-wise Conformer for Streaming ASR Using Sequentially Sampled Chunks and Chunked Causal Convolution21 Nov 2022
- 2022FusionFormer: Fusing Operations in Transformer for Efficient Streaming Speech Recognition31 Oct 2022
- 2022Adaptive Sparse and Monotonic Attention for Transformer-based Automatic Speech Recognition30 Sep 2022
- 2022CUSIDE: Chunking, Simulating Future Context and Decoding for Streaming ASR31 Mar 2022
- 2022Shifted Chunk Encoder for Transformer Based Streaming End-to-End ASR29 Mar 2022
- 2021Recent Advances in End-to-End Automatic Speech Recognition2 Nov 2021
- 2021Streaming End-to-End ASR based on Blockwise Non-Autoregressive Models20 Jul 2021
- 2021U2++: Unified Two-pass Bidirectional End-to-end Model for Speech Recognition10 Jun 2021
- 2021WNARS: WFST based Non-autoregressive Streaming End-to-End Speech Recognition8 Apr 2021
12 newest of 16 All 16 (JSON)