# Multi-Modal Data Augmentation for End-to-End ASR

arXiv: https://arxiv.org/abs/1803.10299 · HTML: https://mlsanity.com/paper/1803.10299/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Mar 2018.

## Built on by (most cited first, 18 of 18)

- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages](https://mlsanity.com/paper/2303.01037/) (2023)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [MAESTRO: Matched Speech Text Representations through Modality Matching](https://mlsanity.com/paper/2204.03409/) (2022)
- [Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task](https://mlsanity.com/paper/2107.05782/) (2021)
- [Leveraging Large Text Corpora for End-to-End Speech Summarization](https://mlsanity.com/paper/2303.00978/) (2023)
- [Code-Switching Text Generation and Injection in Mandarin-English ASR](https://mlsanity.com/paper/2303.10949/) (2023)
- [A Survey on Multimodal Wearable Sensor-based Human Action Recognition](https://mlsanity.com/paper/2404.15349/) (2024)
- [AfriNames: Most ASR models "butcher" African Names](https://mlsanity.com/paper/2306.00253/) (2023)
- [Speech-text based multi-modal training with bidirectional attention for improved speech recognition](https://mlsanity.com/paper/2211.00325/) (2022)
- [Improving Joint Speech-Text Representations Without Alignment](https://mlsanity.com/paper/2308.06125/) (2023)
- [Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharing](https://mlsanity.com/paper/2309.15826/) (2023)
- [Improving noisy student training for low-resource languages in End-to-End ASR using CycleGAN and inter-domain losses](https://mlsanity.com/paper/2407.21061/) (2024)
- [Towards Robust Waveform-Based Acoustic Models](https://mlsanity.com/paper/2110.08634/) (2021)
- [A Complementary Joint Training Approach Using Unpaired Speech and Text for Low-Resource Automatic Speech Recognition](https://mlsanity.com/paper/2204.02023/) (2022)
- [End-to-End Open Vocabulary Keyword Search With Multilingual Neural Representations](https://mlsanity.com/paper/2308.08027/) (2023)
- [Written Term Detection Improves Spoken Term Detection](https://mlsanity.com/paper/2407.04601/) (2024)

## Abstract (arXiv)

We present a new end-to-end architecture for automatic speech recognition (ASR) that can be trained using symbolic input in addition to the traditional acoustic input. This architecture utilizes two separate encoders: one for acoustic input and another for symbolic input, both sharing the attention and decoder parameters. We call this architecture a multi-modal data augmentation network (MMDA), as it can support multi-modal (acoustic and symbolic) input and enables seamless mixing of large text datasets with significantly smaller transcribed speech corpora during training. We study different ways of transforming large text corpora into a symbolic form suitable for training our MMDA network. Our best MMDA setup obtains small improvements on character error rate (CER), and as much as 7-10% relative word error rate (WER) improvement over a baseline both with and without an external language model.