# Translating speech with just images

arXiv: https://arxiv.org/abs/2406.07133 · HTML: https://mlsanity.com/paper/2406.07133/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 37 references that are in the map.
- First posted on arXiv 11 Jun 2024.

## Builds on (most cited first, 29 of 29)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://mlsanity.com/paper/1610.02424/) (2016)
- [SoundNet: Learning Sound Representations from Unlabeled Video](https://mlsanity.com/paper/1610.09001/) (2016)
- [Cross Modal Distillation for Supervision Transfer](https://mlsanity.com/paper/1507.00448/) (2015)
- [AVLnet: Learning Audio-Visual Language Representations from Instructional Videos](https://mlsanity.com/paper/2006.09199/) (2020)
- [Plex: Towards Reliability using Pretrained Large Model Extensions](https://mlsanity.com/paper/2207.07411/) (2022)
- [SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentation](https://mlsanity.com/paper/2209.15323/) (2022)
- [Deep Multimodal Semantic Embeddings for Speech and Images](https://mlsanity.com/paper/1511.03690/) (2015)
- [Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech](https://mlsanity.com/paper/1911.09602/) (2019)
- [MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Prompting](https://mlsanity.com/paper/2210.07179/) (2022)
- [Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques](https://mlsanity.com/paper/2104.13225/) (2021)
- [SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model](https://mlsanity.com/paper/2210.00705/) (2022)
- [Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speech](https://mlsanity.com/paper/1804.03052/) (2018)
- [Semantic speech retrieval with a visually grounded model of untranscribed speech](https://mlsanity.com/paper/1710.01949/) (2017)
- [Visually grounded cross-lingual keyword spotting in speech](https://mlsanity.com/paper/1806.05030/) (2018)
- [Talk, Don't Write: A Study of Direct Speech-Based Image Retrieval](https://mlsanity.com/paper/2104.01894/) (2021)
- [Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer](https://mlsanity.com/paper/2112.08995/) (2021)
- [M-SpeechCLIP: Leveraging Large-Scale, Pre-Trained Models for Multilingual Speech to Image Retrieval](https://mlsanity.com/paper/2211.01180/) (2022)
- [YFACC: A Yor`ub'a speech-image dataset for cross-lingual keyword localisation through visual grounding](https://mlsanity.com/paper/2210.04600/) (2022)
- [Visually grounded few-shot word learning in low-resource settings](https://mlsanity.com/paper/2306.11371/) (2023)

## Built on by (most cited first, 2 of 2)

- [Connecting Speech to Words through Images](https://mlsanity.com/paper/2606.16807/) (2026)
- [Mapping Written Words to Spoken Words in a Different Language Using Only Visual Grounding](https://mlsanity.com/paper/2608.26925/) (2026)

## Abstract (arXiv)

Visually grounded speech models link speech to images. We extend this connection by linking images to text via an existing image captioning system, and as a result gain the ability to map speech audio directly to text. This approach can be used for speech translation with just images by having the audio in a different language from the generated captions. We investigate such a system on a real low-resource language, Yor`ub'a, and propose a Yor`ub'a-to-English speech translation model that leverages pretrained components in order to be able to learn in the low-resource regime. To limit overfitting, we find that it is essential to use a decoding scheme that produces diverse image captions for training. Results show that the predicted translations capture the main semantics of the spoken audio, albeit in a simpler and shorter form.