mlsanityData to 7 Oct 2026

Multi-Modal Data Augmentation for End-to-End ASR

arXiv 1803.10299 ↗First posted 27 Mar 2018cs.CL cs.SD eess.AScited by 18

Not on the map. Why?

Abstract (arXiv)

We present a new end-to-end architecture for automatic speech recognition (ASR) that can be trained using symbolic input in addition to the traditional acoustic input. This architecture utilizes two separate encoders: one for acoustic input and another for symbolic input, both sharing the attention and decoder parameters. We call this architecture a multi-modal data augmentation network (MMDA), as it can support multi-modal (acoustic and symbolic) input and enables seamless mixing of large text datasets with significantly smaller transcribed speech corpora during training. We study different ways of transforming large text corpora into a symbolic form suitable for training our MMDA network. Our best MMDA setup obtains small improvements on character error rate (CER), and as much as 7-10% relative word error rate (WER) improvement over a baseline both with and without an external language model.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 4; 2023: 8; 2024: 3.

18 papers.

Sort:

  1. 2021A Survey of Data Augmentation Approaches for NLPcited by 408
  2. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  3. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  4. 2023End-to-End Speech Recognition: A Surveycited by 127
  5. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  6. 2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
  7. 2023Leveraging Large Text Corpora for End-to-End Speech Summarizationcited by 11
  8. 2023Code-Switching Text Generation and Injection in Mandarin-English ASRcited by 10
  9. 2024A Survey on Multimodal Wearable Sensor-based Human Action Recognitioncited by 10
  10. 2023AfriNames: Most ASR models "butcher" African Namescited by 7
  11. 2022Speech-text based multi-modal training with bidirectional attention for improved speech recognitioncited by 4
  12. 2023Improving Joint Speech-Text Representations Without Alignmentcited by 4
6 more
  1. 2023Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharingcited by 3
  2. 2024Improving noisy student training for low-resource languages in End-to-End ASR using CycleGAN and inter-domain lossescited by 2
  3. 2021Towards Robust Waveform-Based Acoustic Modelscited by 1
  4. 2022A Complementary Joint Training Approach Using Unpaired Speech and Text for Low-Resource Automatic Speech Recognitioncited by 1
  5. 2023End-to-End Open Vocabulary Keyword Search With Multilingual Neural Representationscited by 0
  6. 2024Written Term Detection Improves Spoken Term Detectioncited by 0