# Talk, Don't Write: A Study of Direct Speech-Based Image Retrieval

arXiv: https://arxiv.org/abs/2104.01894 · HTML: https://mlsanity.com/paper/2104.01894/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 5 Apr 2021.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Connecting Vision and Language with Localized Narratives](https://mlsanity.com/paper/1912.03098/) (2019)
- [Visual Semantic Reasoning for Image-Text Matching](https://mlsanity.com/paper/1909.02701/) (2019)
- [AVLnet: Learning Audio-Visual Language Representations from Instructional Videos](https://mlsanity.com/paper/2006.09199/) (2020)
- [From Captions to Visual Concepts and Back](https://mlsanity.com/paper/1411.4952/) (2014)
- [Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input](https://mlsanity.com/paper/1804.01452/) (2018)
- [Deep Multimodal Semantic Embeddings for Speech and Images](https://mlsanity.com/paper/1511.03690/) (2015)
- [End-to-End Retrieval in Continuous Space](https://mlsanity.com/paper/1811.08008/) (2018)
- [Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO](https://mlsanity.com/paper/2004.15020/) (2020)
- [Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech](https://mlsanity.com/paper/1911.09602/) (2019)
- [Representations of language in a model of visually grounded speech signal](https://mlsanity.com/paper/1702.01991/) (2017)
- [Large-scale representation learning from visually grounded untranscribed speech](https://mlsanity.com/paper/1909.08782/) (2019)
- [Learning Word-Like Units from Joint Audio-Visual Analysis](https://mlsanity.com/paper/1701.07481/) (2017)
- [Language learning using Speech to Image retrieval](https://mlsanity.com/paper/1909.03795/) (2019)
- [Visually grounded cross-lingual keyword spotting in speech](https://mlsanity.com/paper/1806.05030/) (2018)
- [Visually grounded learning of keyword prediction from untranscribed speech](https://mlsanity.com/paper/1703.08136/) (2017)
- [Towards Visually Grounded Sub-Word Speech Unit Discovery](https://mlsanity.com/paper/1902.08213/) (2019)
- [Word Recognition, Competition, and Activation in a Model of Visually Grounded Speech](https://mlsanity.com/paper/1909.08491/) (2019)
- [Symbolic inductive bias for visually grounded learning of spoken language](https://mlsanity.com/paper/1812.09244/) (2018)
- [Catplayinginthesnow: Impact of Prior Segmentation on a Model of Visually Grounded Speech](https://mlsanity.com/paper/2006.08387/) (2020)
- [Textual Supervision for Visually Grounded Spoken Language Understanding](https://mlsanity.com/paper/2010.02806/) (2020)
- [From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learning](https://mlsanity.com/paper/1610.03342/) (2016)
- [Learning semantic sentence representations from visually grounded language without lexical knowledge](https://mlsanity.com/paper/1903.11393/) (2019)

## Built on by (most cited first, 19 of 19)

- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition](https://mlsanity.com/paper/2106.05933/) (2021)
- [Cross-Modal Discrete Representation Learning](https://mlsanity.com/paper/2106.05438/) (2021)
- [Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques](https://mlsanity.com/paper/2104.13225/) (2021)
- [SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model](https://mlsanity.com/paper/2210.00705/) (2022)
- [Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling](https://mlsanity.com/paper/2202.03543/) (2022)
- [Fast-Slow Transformer for Visually Grounding Speech](https://mlsanity.com/paper/2109.08186/) (2021)
- [TVLT: Textless Vision-Language Transformer](https://mlsanity.com/paper/2209.14156/) (2022)
- [Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Model](https://mlsanity.com/paper/2305.11435/) (2023)
- [Cascaded Multilingual Audio-Visual Learning from Videos](https://mlsanity.com/paper/2111.04823/) (2021)
- [ConceptBeam: Concept Driven Target Speech Extraction](https://mlsanity.com/paper/2207.11964/) (2022)
- [You Only Speak Once to See](https://mlsanity.com/paper/2409.18372/) (2024)
- [Spoken ObjectNet: A Bias-Controlled Spoken Caption Dataset](https://mlsanity.com/paper/2110.07575/) (2021)
- [Translating speech with just images](https://mlsanity.com/paper/2406.07133/) (2024)
- [Integrating Self-supervised Speech Model with Pseudo Word-level Targets from Visually-grounded Speech Model](https://mlsanity.com/paper/2402.05819/) (2024)
- [Coarse-to-fine Alignment Makes Better Speech-image Retrieval](https://mlsanity.com/paper/2408.13119/) (2024)
- [Cross-Modal Coordination Across a Diverse Set of Input Modalities](https://mlsanity.com/paper/2401.16347/) (2024)
- [BrewCLIP: A Bifurcated Representation Learning Framework for Audio-Visual Retrieval](https://mlsanity.com/paper/2408.10383/) (2024)
- [Cross-Modal Denoising: A Novel Training Paradigm for Enhancing Speech-Image Retrieval](https://mlsanity.com/paper/2408.13705/) (2024)

## Abstract (arXiv)

Speech-based image retrieval has been studied as a proxy for joint representation learning, usually without emphasis on retrieval itself. As such, it is unclear how well speech-based retrieval can work in practice -- both in an absolute sense and versus alternative strategies that combine automatic speech recognition (ASR) with strong text encoders. In this work, we extensively study and expand choices of encoder architectures, training methodology (including unimodal and multimodal pretraining), and other factors. Our experiments cover different types of speech in three datasets: Flickr Audio, Places Audio, and Localized Narratives. Our best model configuration achieves large gains over state of the art, e.g., pushing recall-at-one from 21.8% to 33.2% for Flickr Audio and 27.6% to 53.4% for Places Audio. We also show our best speech-based models can match or exceed cascaded ASR-to-text encoding when speech is spontaneous, accented, or otherwise hard to automatically transcribe.