mlsanityData to 7 Oct 2026

Talk, Don't Write: A Study of Direct Speech-Based Image Retrieval

arXiv 2104.01894 ↗First posted 5 Apr 2021cs.CL cs.CV cs.IRcited by 19

Not on the map. Why?

Abstract (arXiv)

Speech-based image retrieval has been studied as a proxy for joint representation learning, usually without emphasis on retrieval itself. As such, it is unclear how well speech-based retrieval can work in practice -- both in an absolute sense and versus alternative strategies that combine automatic speech recognition (ASR) with strong text encoders. In this work, we extensively study and expand choices of encoder architectures, training methodology (including unimodal and multimodal pretraining), and other factors. Our experiments cover different types of speech in three datasets: Flickr Audio, Places Audio, and Localized Narratives. Our best model configuration achieves large gains over state of the art, e.g., pushing recall-at-one from 21.8% to 33.2% for Flickr Audio and 27.6% to 53.4% for Places Audio. We also show our best speech-based models can match or exceed cascaded ASR-to-text encoding when speech is spontaneous, accented, or otherwise hard to automatically transcribe.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 26 of its 35 references that are in the map. First posted on arXiv 5 Apr 2021.

Builds on

26 of 35 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  5. 2019Connecting Vision and Language with Localized Narrativescited by 225
  6. 2019Visual Semantic Reasoning for Image-Text Matchingcited by 185
  7. 2020AVLnet: Learning Audio-Visual Language Representations from Instructional Videoscited by 110
  8. 2014From Captions to Visual Concepts and Backcited by 109
  9. 2018Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Inputcited by 81
  10. 2015Deep Multimodal Semantic Embeddings for Speech and Imagescited by 56
  11. 2018End-to-End Retrieval in Continuous Spacecited by 46
  12. 2020Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCOcited by 46
14 more
  1. 2019Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speechcited by 44
  2. 2017Representations of language in a model of visually grounded speech signalcited by 37
  3. 2019Large-scale representation learning from visually grounded untranscribed speechcited by 33
  4. 2017Learning Word-Like Units from Joint Audio-Visual Analysiscited by 27
  5. 2019Language learning using Speech to Image retrievalcited by 21
  6. 2018Visually grounded cross-lingual keyword spotting in speechcited by 19
  7. 2017Visually grounded learning of keyword prediction from untranscribed speechcited by 16
  8. 2019Towards Visually Grounded Sub-Word Speech Unit Discoverycited by 15
  9. 2019Word Recognition, Competition, and Activation in a Model of Visually Grounded Speechcited by 13
  10. 2018Symbolic inductive bias for visually grounded learning of spoken languagecited by 10
  11. 2020Catplayinginthesnow: Impact of Prior Segmentation on a Model of Visually Grounded Speechcited by 9
  12. 2020Textual Supervision for Visually Grounded Spoken Language Understandingcited by 9
  13. 2016From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learningcited by 8
  14. 2019Learning semantic sentence representations from visually grounded language without lexical knowledgecited by 4

Built on by

Cited by year

Citing papers per year: 2021: 6; 2022: 5; 2023: 1; 2024: 7.

19 papers.

Sort:

  1. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  2. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  3. 2021Cross-Modal Discrete Representation Learningcited by 35
  4. 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniquescited by 32
  5. 2022SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Modelcited by 29
  6. 2022Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modelingcited by 25
  7. 2021Fast-Slow Transformer for Visually Grounding Speechcited by 23
  8. 2022TVLT: Textless Vision-Language Transformercited by 22
  9. 2023Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Modelcited by 11
  10. 2021Cascaded Multilingual Audio-Visual Learning from Videoscited by 8
  11. 2022ConceptBeam: Concept Driven Target Speech Extractioncited by 8
  12. 2024You Only Speak Once to Seecited by 4
7 more
  1. 2021Spoken ObjectNet: A Bias-Controlled Spoken Caption Datasetcited by 2
  2. 2024Translating speech with just imagescited by 2
  3. 2024Integrating Self-supervised Speech Model with Pseudo Word-level Targets from Visually-grounded Speech Modelcited by 1
  4. 2024Coarse-to-fine Alignment Makes Better Speech-image Retrievalcited by 1
  5. 2024Cross-Modal Coordination Across a Diverse Set of Input Modalitiescited by 0
  6. 2024BrewCLIP: A Bifurcated Representation Learning Framework for Audio-Visual Retrievalcited by 0
  7. 2024Cross-Modal Denoising: A Novel Training Paradigm for Enhancing Speech-Image Retrievalcited by 0