mlsanityData to 7 Oct 2026

Fast-Slow Transformer for Visually Grounding Speech

arXiv 2109.08186 ↗First posted 16 Sep 2021eess.AS cs.CL cs.IRcited by 23

Not on the map. Why?

Abstract (arXiv)

We present Fast-Slow Transformer for Visually Grounding Speech, or FaST-VGS. FaST-VGS is a Transformer-based model for learning the associations between raw speech waveforms and visual images. The model unifies dual-encoder and cross-attention architectures into a single model, reaping the superior retrieval speed of the former along with the accuracy of the latter. FaST-VGS achieves state-of-the-art speech-image retrieval accuracy on benchmark datasets, and its learned representations exhibit strong performance on the ZeroSpeech 2021 phonetic and semantic tasks.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 35 of its 46 references that are in the map. First posted on arXiv 16 Sep 2021.

Builds on

35 of 46 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  10. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  11. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 100
  12. 2021Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformerscited by 93
23 more
  1. 2018Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Inputcited by 81
  2. 2015Deep Multimodal Semantic Embeddings for Speech and Imagescited by 56
  3. 2020Text-Free Image-to-Speech Synthesis Using Learned Segmental Unitscited by 50
  4. 2019Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speechcited by 44
  5. 2017Representations of language in a model of visually grounded speech signalcited by 37
  6. 2019Large-scale representation learning from visually grounded untranscribed speechcited by 33
  7. 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniquescited by 32
  8. 2018Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speechcited by 26
  9. 2021Analyzing Speaker Information in Self-Supervised Models to Improve Zero-Resource Speech Processingcited by 26
  10. 2019Language learning using Speech to Image retrievalcited by 21
  11. 2018Visually grounded cross-lingual keyword spotting in speechcited by 19
  12. 2021Talk, Don't Write: A Study of Direct Speech-Based Image Retrievalcited by 19
  13. 2017Visually grounded learning of keyword prediction from untranscribed speechcited by 16
  14. 2019Transfer Learning from Audio-Visual Grounding to Speech Recognitioncited by 16
  15. 2019Towards Visually Grounded Sub-Word Speech Unit Discoverycited by 15
  16. 2021Attention-Based Keyword Localisation in Speech using Visual Groundingcited by 13
  17. 2018Symbolic inductive bias for visually grounded learning of spoken languagecited by 10
  18. 2018End-to-End Multimodal Speech Recognitioncited by 9
  19. 2020Catplayinginthesnow: Impact of Prior Segmentation on a Model of Visually Grounded Speechcited by 9
  20. 2020Textual Supervision for Visually Grounded Spoken Language Understandingcited by 9
  21. 2021Evaluation of Audio-Visual Alignments in Visually Grounded Speech Modelscited by 8
  22. 2021Information Retrieval for ZeroSpeech 2021: The Submission by University of Wroclawcited by 7
  23. 2021ZR-2021VG: Zero-Resource Speech Challenge, Visually-Grounded Language Modelling track, 2021 editioncited by 6

Built on by

Cited by year

Citing papers per year: 2022: 10; 2023: 3; 2024: 9; 2025: 1.

23 papers.

Sort:

  1. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  2. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 155
  3. 2022Word Discovery in Visually Grounded, Self-Supervised Speech Modelscited by 37
  4. 2022SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Modelcited by 29
  5. 2022Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modelingcited by 25
  6. 2023Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Modelcited by 11
  7. 2022M-SpeechCLIP: Leveraging Large-Scale, Pre-Trained Models for Multilingual Speech to Image Retrievalcited by 9
  8. 2022Learning English with Peppa Pigcited by 8
  9. 2022ConceptBeam: Concept Driven Target Speech Extractioncited by 8
  10. 2023What a Whole Slide Image Can Tell? Subtype-guided Masked Transformer for Pathological Image Captioningcited by 8
  11. 2022Keyword localisation in untranscribed speech using visually grounded speech modelscited by 7
  12. 2022Towards visually prompted keyword localisation for zero-resource spoken languagescited by 6
11 more
  1. 2024Interface Design for Self-Supervised Speech Modelscited by 5
  2. 2024Visually Grounded Speech Models have a Mutual Exclusivity Biascited by 3
  3. 2024A model of early word acquisition based on realistic-scale audiovisual naming eventscited by 3
  4. 2024Measuring Sound Symbolism in Audio-visual Modelscited by 3
  5. 2023Simultaneous or Sequential Training? How Speech Representations Cooperate in a Multi-Task Self-Supervised Learning Systemcited by 2
  6. 2024SpeechCLIP+: Self-supervised multi-task representation learning for speech via CLIP and speech-image datacited by 2
  7. 2024Integrating Self-supervised Speech Model with Pseudo Word-level Targets from Visually-grounded Speech Modelcited by 1
  8. 2024Coarse-to-fine Alignment Makes Better Speech-image Retrievalcited by 1
  9. 2025The Curious Case of Visual Grounding: Different Effects for Speech- and Text-based Language Encoderscited by 1
  10. 2024Cross-Modal Denoising: A Novel Training Paradigm for Enhancing Speech-Image Retrievalcited by 0
  11. 2024Improved Visually Prompted Keyword Localisation in Real Low-Resource Settingscited by 0