mlsanityData to 7 Oct 2026

Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling

arXiv 2202.03543 ↗First posted 7 Feb 2022eess.AS cs.CL cs.CVcited by 25

Not on the map. Why?

Abstract (arXiv)

In this paper, we describe our submissions to the ZeroSpeech 2021 Challenge and SUPERB benchmark. Our submissions are based on the recently proposed FaST-VGS model, which is a Transformer-based model that learns to associate raw speech waveforms with semantically related images, all without the use of any transcriptions of the speech. Additionally, we introduce a novel extension of this model, FaST-VGS+, which is learned in a multi-task fashion with a masked language modeling objective in addition to the visual grounding objective. On ZeroSpeech 2021, we show that our models perform competitively on the ABX task, outperform all other concurrent submissions on the Syntactic and Semantic tasks, and nearly match the best system on the Lexical task. On the SUPERB benchmark, we show that our models also achieve strong performance, in some cases even outperforming the popular wav2vec2.0 model.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 56 of its 64 references that are in the map. First posted on arXiv 7 Feb 2022.

Builds on

56 of 64 references.

References · 56

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  10. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  12. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
38 more
  1. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  2. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  3. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  4. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  5. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  6. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
  7. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  8. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  9. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
  10. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 100
  11. 2020Exploring wav2vec 2.0 on speaker verification and language identificationcited by 93
  12. 2019Learning Problem-agnostic Speech Representations from Multiple Self-supervised Taskscited by 87
  13. 2018Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Inputcited by 81
  14. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  15. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  16. 2020Vector-Quantized Autoregressive Predictive Codingcited by 69
  17. 2015Deep Multimodal Semantic Embeddings for Speech and Imagescited by 56
  18. 2020Text-Free Image-to-Speech Synthesis Using Learned Segmental Unitscited by 50
  19. 2019Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speechcited by 44
  20. 2017Representations of language in a model of visually grounded speech signalcited by 37
  21. 2020FragmentVC: Any-to-Any Voice Conversion by End-to-End Extracting and Fusing Fine-Grained Voice Fragments With Attentioncited by 37
  22. 2020Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretrainingcited by 35
  23. 2019Large-scale representation learning from visually grounded untranscribed speechcited by 33
  24. 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniquescited by 32
  25. 2017Learning Word-Like Units from Joint Audio-Visual Analysiscited by 27
  26. 2018Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speechcited by 26
  27. 2021Analyzing Speaker Information in Self-Supervised Models to Improve Zero-Resource Speech Processingcited by 26
  28. 2021Fast-Slow Transformer for Visually Grounding Speechcited by 23
  29. 2019Language learning using Speech to Image retrievalcited by 21
  30. 2018Visually grounded cross-lingual keyword spotting in speechcited by 19
  31. 2021Talk, Don't Write: A Study of Direct Speech-Based Image Retrievalcited by 19
  32. 2017Visually grounded learning of keyword prediction from untranscribed speechcited by 16
  33. 2019Transfer Learning from Audio-Visual Grounding to Speech Recognitioncited by 16
  34. 2019Towards Visually Grounded Sub-Word Speech Unit Discoverycited by 15
  35. 2021Attention-Based Keyword Localisation in Speech using Visual Groundingcited by 13
  36. 2018Symbolic inductive bias for visually grounded learning of spoken languagecited by 10
  37. 2018End-to-End Multimodal Speech Recognitioncited by 9
  38. 2020Catplayinginthesnow: Impact of Prior Segmentation on a Model of Visually Grounded Speechcited by 9

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 4; 2024: 9; 2025: 2; 2026: 1.

25 papers.

Sort:

  1. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  2. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  3. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 155
  4. 2023What Do Self-Supervised Speech Models Know About Words?cited by 62
  5. 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 38
  6. 2022Word Discovery in Visually Grounded, Self-Supervised Speech Modelscited by 37
  7. 2022Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challengecited by 34
  8. 2022SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Modelcited by 29
  9. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  10. 2023Wave to Syntax: Probing spoken language models for syntaxcited by 18
  11. 2024Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding of Sound and Languagecited by 17
  12. 2022Evaluating context-invariance in unsupervised speech representationscited by 11
13 more
  1. 2023Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Modelcited by 11
  2. 2022ConceptBeam: Concept Driven Target Speech Extractioncited by 8
  3. 2024Visually Grounded Speech Models have a Mutual Exclusivity Biascited by 3
  4. 2024A model of early word acquisition based on realistic-scale audiovisual naming eventscited by 3
  5. 2023Simultaneous or Sequential Training? How Speech Representations Cooperate in a Multi-Task Self-Supervised Learning Systemcited by 2
  6. 2024SpeechCLIP+: Self-supervised multi-task representation learning for speech via CLIP and speech-image datacited by 2
  7. 2025The Curious Case of Visual Grounding: Different Effects for Speech- and Text-based Language Encoderscited by 1
  8. 2025Leveraging Audio-Visual Data to Reduce the Multilingual Gap in Self-Supervised Speech Modelscited by 1
  9. 2026Computational modeling of early language learning from acoustic speech and audiovisual input without linguistic priorscited by 1
  10. 2024Learning Semantic Information from Raw Audio Signal Using Both Contextual and Phonetic Representationscited by 0
  11. 2024Multimodal Input Aids a Bayesian Model of Phonetic Learningcited by 0
  12. 2024Cross-Modal Denoising: A Novel Training Paradigm for Enhancing Speech-Image Retrievalcited by 0
  13. 2024Visually Grounded Speech Models for Low-resource Languages and Cognitive Modellingcited by 0