# Augmenting Librispeech with French Translations: A Multimodal Corpus for Direct Speech Translation Evaluation

arXiv: https://arxiv.org/abs/1802.03142 · HTML: https://mlsanity.com/paper/1802.03142/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Feb 2018.

## Built on by (most cited first, 30 of 34)

- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [The Multilingual TEDx Corpus for Speech Recognition and Translation](https://mlsanity.com/paper/2102.01757/) (2021)
- [Learning Shared Semantic Space for Speech-to-Text Translation](https://mlsanity.com/paper/2105.03095/) (2021)
- [Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders](https://mlsanity.com/paper/2105.05752/) (2021)
- [Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translation](https://mlsanity.com/paper/2102.05766/) (2021)
- [End-to-end Speech Translation via Cross-modal Progressive Training](https://mlsanity.com/paper/2104.10380/) (2021)
- [Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization](https://mlsanity.com/paper/2305.11095/) (2023)
- [RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformer](https://mlsanity.com/paper/2106.04833/) (2021)
- [MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation](https://mlsanity.com/paper/2303.00628/) (2023)
- [Revisiting End-to-End Speech-to-Text Translation From Scratch](https://mlsanity.com/paper/2206.04571/) (2022)
- [BSTC: A Large-Scale Chinese-English Speech Translation Dataset](https://mlsanity.com/paper/2104.03575/) (2021)
- [Non-Parametric Domain Adaptation for End-to-End Speech Translation](https://mlsanity.com/paper/2205.11211/) (2022)
- [Simple and Effective Unsupervised Speech Translation](https://mlsanity.com/paper/2210.10191/) (2022)
- [Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoring](https://mlsanity.com/paper/2109.04411/) (2021)
- [Fast-MD: Fast Multi-Decoder End-to-End Speech Translation with Non-Autoregressive Hidden Intermediates](https://mlsanity.com/paper/2109.12804/) (2021)
- [Improving End-to-end Speech Translation by Leveraging Auxiliary Speech and Text Data](https://mlsanity.com/paper/2212.01778/) (2022)
- [End-to-End Speech-to-Text Translation: A Survey](https://mlsanity.com/paper/2312.01053/) (2023)
- [ESPnet-SE++: Speech Enhancement for Robust Speech Recognition, Translation, and Understanding](https://mlsanity.com/paper/2207.09514/) (2022)
- [Kosp2e: Korean Speech to English Translation Corpus](https://mlsanity.com/paper/2107.02875/) (2021)
- [SpliceOut: A Simple and Efficient Audio Augmentation Method](https://mlsanity.com/paper/2110.00046/) (2021)
- [Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learning](https://mlsanity.com/paper/2304.10309/) (2023)
- [On the Impact of Noises in Crowd-Sourced Data for Speech Translation](https://mlsanity.com/paper/2206.13756/) (2022)
- [Does Joint Training Really Help Cascaded Speech Translation?](https://mlsanity.com/paper/2210.13700/) (2022)
- [HK-LegiCoST: Leveraging Non-Verbatim Transcripts for Speech Translation](https://mlsanity.com/paper/2306.11252/) (2023)
- [FFSTC: Fongbe to French Speech Translation Corpus](https://mlsanity.com/paper/2403.05488/) (2024)
- [A High-Quality and Large-Scale Dataset for English-Vietnamese Speech Translation](https://mlsanity.com/paper/2208.04243/) (2022)
- [End-to-End Single-Channel Speaker-Turn Aware Conversational Speech Translation](https://mlsanity.com/paper/2311.00697/) (2023)
- [LibriS2S: A German-English Speech-to-Speech Translation Corpus](https://mlsanity.com/paper/2204.10593/) (2022)
- [DropDim: A Regularization Method for Transformer Networks](https://mlsanity.com/paper/2304.10321/) (2023)
- [Conditioning LLMs with Emotion in Neural Machine Translation](https://mlsanity.com/paper/2408.03150/) (2024)

## Abstract (arXiv)

Recent works in spoken language translation (SLT) have attempted to build end-to-end speech-to-text translation without using source language transcription during learning or decoding. However, while large quantities of parallel texts (such as Europarl, OpenSubtitles) are available for training machine translation systems, there are no large (100h) and open source parallel corpora that include speech in a source language aligned to text in a target language. This paper tries to fill this gap by augmenting an existing (monolingual) corpus: LibriSpeech. This corpus, used for automatic speech recognition, is derived from read audiobooks from the LibriVox project, and has been carefully segmented and aligned. After gathering French e-books corresponding to the English audio-books from LibriSpeech, we align speech segments at the sentence level with their respective translations and obtain 236h of usable parallel data. This paper presents the details of the processing as well as a manual evaluation conducted on a small subset of the corpus. This evaluation shows that the automatic alignments scores are reasonably correlated with the human judgments of the bilingual alignment quality. We believe that this corpus (which is made available online) is useful for replicable experiments in direct speech translation or more general spoken language translation experiments.