# Towards speech-to-text translation without speech recognition

arXiv: https://arxiv.org/abs/1702.03856 · HTML: https://mlsanity.com/paper/1702.03856/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Feb 2017.

## Built on by (most cited first, 11 of 11)

- [T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translation](https://mlsanity.com/paper/2205.12216/) (2022)
- [Simple and Effective Unsupervised Speech Translation](https://mlsanity.com/paper/2210.10191/) (2022)
- [A Brief Overview of Unsupervised Neural Speech Representation Learning](https://mlsanity.com/paper/2203.01829/) (2022)
- [End-to-End Speech-to-Text Translation: A Survey](https://mlsanity.com/paper/2312.01053/) (2023)
- [YFACC: A Yor`ub'a speech-image dataset for cross-lingual keyword localisation through visual grounding](https://mlsanity.com/paper/2210.04600/) (2022)
- [Improving Speech-to-Speech Translation Through Unlabeled Text](https://mlsanity.com/paper/2210.14514/) (2022)
- [Speech-to-SQL: Towards Speech-driven SQL Query Generation From Natural Language Question](https://mlsanity.com/paper/2201.01209/) (2022)
- [Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparison](https://mlsanity.com/paper/2501.02370/) (2025)
- [Prabhupadavani: A Code-mixed Speech Translation Data for 25 Languages](https://mlsanity.com/paper/2201.11391/) (2022)
- [Modular Speech-to-Text Translation for Zero-Shot Cross-Modal Transfer](https://mlsanity.com/paper/2310.03724/) (2023)
- [Learning Phone Recognition from Unpaired Audio and Phone Sequences Based on Generative Adversarial Network](https://mlsanity.com/paper/2207.14568/) (2022)

## Abstract (arXiv)

We explore the problem of translating speech to text in low-resource scenarios where neither automatic speech recognition (ASR) nor machine translation (MT) are available, but we have training data in the form of audio paired with text translations. We present the first system for this problem applied to a realistic multi-speaker dataset, the CALLHOME Spanish-English speech translation corpus. Our approach uses unsupervised term discovery (UTD) to cluster repeated patterns in the audio, creating a pseudotext, which we pair with translations to create a parallel text and train a simple bag-of-words MT model. We identify the challenges faced by the system, finding that the difficulty of cross-speaker UTD results in low recall, but that our system is still able to correctly translate some content words in test data.