mlsanityData to 7 Oct 2026

U2++: Unified Two-pass Bidirectional End-to-end Model for Speech Recognition

arXiv 2106.05642 ↗First posted 10 Jun 2021cs.SD cs.CL eess.AScited by 28

Not on the map. Why?

Abstract (arXiv)

The unified streaming and non-streaming two-pass (U2) end-to-end model for speech recognition has shown great performance in terms of streaming capability, accuracy, real-time factor (RTF), and latency. In this paper, we present U2++, an enhanced version of U2 to further improve the accuracy. The core idea of U2++ is to use the forward and the backward information of the labeling sequences at the same time at training to learn richer information, and combine the forward and backward prediction at decoding to give more accurate recognition results. We also proposed a new data augmentation method called SpecSub to help the U2++ model to be more accurate and robust. Our experiments show that, compared with U2, U2++ shows faster convergence at training, better robustness to the decoding method, as well as consistent 5% - 8% word error rate reduction gain over U2. On the experiment of AISHELL-1, we achieve a 4.63% character error rate (CER) with a non-streaming setup and 5.05% with a streaming setup with 320ms latency by U2++. To the best of our knowledge, 5.05% is the best-published streaming result on the AISHELL-1 test set.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 17 of its 22 references that are in the map. First posted on arXiv 10 Jun 2021.

Builds on

17 of 22 references.

References · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  3. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  4. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  5. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
  6. 2015Attention-Based Models for Speech Recognitioncited by 400
  7. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
  8. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
  9. 2021WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkitcited by 123
  10. 2020Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predictcited by 75
  11. 2014End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Resultscited by 68
  12. 2020Unified Streaming and Non-streaming Two-pass End-to-end Model for Speech Recognitioncited by 37
5 more
  1. 2019Synchronous Transformers for End-to-End Speech Recognitioncited by 24
  2. 2020Enhancing Monotonic Multihead Attention for Streaming ASRcited by 16
  3. 2020Streaming Chunk-Aware Multihead Attention for Online End-to-End Speech Recognitioncited by 12
  4. 2021WNARS: WFST based Non-autoregressive Streaming End-to-End Speech Recognitioncited by 10
  5. 2020Transformer with Bidirectional Decoder for Speech Recognitioncited by 7

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 8; 2023: 9; 2024: 6; 2025: 4.

28 papers.

Sort:

By area:

  1. 2023End-to-End Speech Recognition: A Surveycited by 127
  2. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  3. 2024A Full-duplex Speech Dialogue Scheme Based On Large Language Modelscited by 40
  4. 2023Leveraging Large Language Models for Exploiting ASR Uncertaintycited by 15
  5. 2024Unveiling the Potential of LLM-Based ASR on Chinese Open-Source Datasetscited by 13
  6. 2022CUSIDE: Chunking, Simulating Future Context and Decoding for Streaming ASRcited by 11
  7. 2023Simultaneous Machine Translation with Large Language Modelscited by 9
  8. 2023Importance of Smoothness Induced by Optimizers in FL4ASR: Towards Understanding Federated Learning for End-to-End ASRcited by 8
  9. 2025Fewer Hallucinations, More Verification: A Three-Stage LLM-Based Framework for ASR Error Correctioncited by 7
  10. 2022TrimTail: Low-Latency Streaming ASR with Simple but Effective Spectrogram-Level Length Penaltycited by 6
  11. 2023Balanced and Explainable Social Media Analysis for Public Health with Large Language Modelscited by 6
  12. 2024SC-MoE: Switch Conformer Mixture of Experts for Unified Streaming and Non-streaming Code-Switching ASRcited by 6
16 more
  1. 2021Non-autoregressive Transformer with Unified Bidirectional Decoder for Automatic Speech Recognitioncited by 5
  2. 2022Audio-to-Intent Using Acoustic-Textual Subword Representations from End-to-End ASRcited by 4
  3. 2022The RoyalFlush System of Speech Recognition for M2MeT Challengecited by 3
  4. 2022Shifted Chunk Encoder for Transformer Based Streaming End-to-End ASRcited by 2
  5. 2022Monolingual Recognizers Fusion for Code-switching Speech Recognitioncited by 2
  6. 2023Multi-Loss Convolutional Network with Time-Frequency Attention for Speech Enhancementcited by 2
  7. 2024UCorrect: An Unsupervised Framework for Automatic Speech Recognition Error Correctioncited by 2
  8. 2025A Unified Speech LLM for Diarization and Speech Recognition in Multilingual Conversationscited by 2
  9. 2022SSCFormer: Push the Limit of Chunk-wise Conformer for Streaming ASR Using Sequentially Sampled Chunks and Chunked Causal Convolutioncited by 1
  10. 2024Device-Directed Speech Detection for Follow-up Conversations Using Large Language Modelscited by 1
  11. 2025CNVSRC 2024: The Second Chinese Continuous Visual Speech Recognition Challengecited by 1
  12. 2025Unifying Streaming and Non-streaming Zipformer-based ASRcited by 1
  13. 2022FusionFormer: Fusing Operations in Transformer for Efficient Streaming Speech Recognitioncited by 0
  14. 2023Enhancing the Unified Streaming and Non-streaming Model with Contrastive Learningcited by 0
  15. 2023ApproBiVT: Lead ASR Models to Generalize Better Using Approximated Bias-Variance Tradeoff Guided Early Stopping and Checkpoint Averagingcited by 0
  16. 2023The GUA-Speech System Description for CNVSRC Challenge 2023cited by 0