mlsanityData to 7 Oct 2026

A Comparative Study on E-Branchformer vs Conformer in Speech Recognition, Translation, and Understanding Tasks

arXiv 2305.11073 ↗First posted 18 May 2023cs.CL cs.SD eess.AScited by 13

Not on the map. Why?

Abstract (arXiv)

Conformer, a convolution-augmented Transformer variant, has become the de facto encoder architecture for speech processing due to its superior performance in various tasks, including automatic speech recognition (ASR), speech translation (ST) and spoken language understanding (SLU). Recently, a new encoder called E-Branchformer has outperformed Conformer in the LibriSpeech ASR benchmark, making it promising for more general speech applications. This work compares E-Branchformer and Conformer through extensive experiments using different types of end-to-end sequence-to-sequence models. Results demonstrate that E-Branchformer achieves comparable or better performance than Conformer in almost all evaluation sets across 15 ASR, 2 ST, and 3 SLU benchmarks, while being more stable during training. We will release our training configurations and pre-trained models for reproducibility, which can benefit the speech community.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 42 of its 58 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

42 of 58 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2016Layer Normalizationcited by 5,579
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  9. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  10. 2016Deep Networks with Stochastic Depthcited by 973
  11. 2016Language Modeling with Gated Convolutional Networkscited by 688
  12. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
30 more
  1. 2015Listen, Attend and Spellcited by 542
  2. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  3. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  4. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
  5. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  6. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
  7. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  8. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 210
  9. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 197
  10. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  11. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  12. 2020Recent Developments on ESPnet Toolkit Boosted by Conformercited by 132
  13. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 108
  14. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 108
  15. 2020ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Contextcited by 96
  16. 2021Intermediate Loss Regularization for CTC-based Speech Recognitioncited by 95
  17. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 90
  18. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 90
  19. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
  20. 2019Jasper: An End-to-End Convolutional Neural Acoustic Modelcited by 75
  21. 2021SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speechcited by 70
  22. 2021Relaxing the Conditional Independence Assumption of CTC-based ASR by Conditioning on Intermediate Predictionscited by 61
  23. 2017JSUT corpus: free large-scale Japanese speech corpus for end-to-end speech synthesiscited by 57
  24. 2021ESPnet-SLU: Advancing Spoken Language Understanding through ESPnetcited by 53
  25. 2018Improved training of end-to-end attention models for speech recognitioncited by 35
  26. 2023Improving Massively Multilingual ASR With Auxiliary CTC Objectivescited by 31
  27. 2022CTC Alignments Improve Autoregressive Translationcited by 25
  28. 2022STOP: A dataset for Spoken Task Oriented Semantic Parsingcited by 22
  29. 2022A Study on the Integration of Pre-trained SSL, ASR, LM and SLU Models for Spoken Language Understandingcited by 18
  30. 2013Improvements to deep convolutional neural networks for LVCSRcited by 11

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 8; 2025: 1.

13 papers.

Sort:

By area:

  1. 2024OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformercited by 74
  2. 2024Towards Robust Speech Representation Learning for Thousands of Languagescited by 48
  3. 2023Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Datacited by 42
  4. 2024OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identificationcited by 34
  5. 2025OWSM v4: Improving Open Whisper-Style Speech Models via Data Scaling and Cleaningcited by 27
  6. 2024ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasetscited by 26
  7. 2024Exploring the Capability of Mamba in Speech Applicationscited by 19
  8. 2024Mamba-based Decoder-Only Approach with Bidirectional Speech Modeling for Speech Recognitioncited by 5
  9. 2023Seq2seq for Automatic Paraphasia Detection in Aphasic Speechcited by 4
  10. 2023Unimodal Aggregation for CTC-based Speech Recognitioncited by 2
  11. 2023End-to-End Single-Channel Speaker-Turn Aware Conversational Speech Translationcited by 2
  12. 2024Multi-Convformer: Extending Conformer with Multiple Convolution Kernelscited by 2
1 more
  1. 2024Compact Speech Translation Models via Discrete Speech Units Pretrainingcited by 1