Translating speech with just images
Not on the map. Why?
Abstract (arXiv)
Visually grounded speech models link speech to images. We extend this connection by linking images to text via an existing image captioning system, and as a result gain the ability to map speech audio directly to text. This approach can be used for speech translation with just images by having the audio in a different language from the generated captions. We investigate such a system on a real low-resource language, Yor`ub'a, and propose a Yor`ub'a-to-English speech translation model that leverages pretrained components in order to be able to learn in the low-resource regime. To limit overfitting, we find that it is essential to use a decoding scheme that produces diverse image captions for training. Results show that the predicted translations capture the main semantics of the spoken audio, albeit in a simpler and shorter form.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 29 of its 37 references that are in the map. First posted on arXiv 11 Jun 2024.
Builds on
29 of 37 references.
References · 29
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
- 2016Sequence-Level Knowledge Distillationcited by 761
- 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
- 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
- 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
- 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
- 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 299
- 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 244
17 more
- 2015Cross Modal Distillation for Supervision Transfercited by 173
- 2020AVLnet: Learning Audio-Visual Language Representations from Instructional Videoscited by 110
- 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
- 2022SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentationcited by 82
- 2015Deep Multimodal Semantic Embeddings for Speech and Imagescited by 56
- 2019Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speechcited by 44
- 2022MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Promptingcited by 41
- 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniquescited by 32
- 2022SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Modelcited by 29
- 2018Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speechcited by 26
- 2017Semantic speech retrieval with a visually grounded model of untranscribed speechcited by 20
- 2018Visually grounded cross-lingual keyword spotting in speechcited by 19
- 2021Talk, Don't Write: A Study of Direct Speech-Based Image Retrievalcited by 19
- 2021Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfercited by 17
- 2022M-SpeechCLIP: Leveraging Large-Scale, Pre-Trained Models for Multilingual Speech to Image Retrievalcited by 9
- 2022YFACC: A Yor`ub'a speech-image dataset for cross-lingual keyword localisation through visual groundingcited by 8
- 2023Visually grounded few-shot word learning in low-resource settingscited by 4
Built on by
Citing papers per year: 2026: 2.
2 papers.