mlsanityData to 14 Oct 2026

Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input

arXiv 1804.01452 ↗First posted 4 Apr 2018cs.CV cs.CL cs.SDcited by 81

Not on the map. Why?

Abstract (arXiv)

In this paper, we explore neural network models that learn to associate segments of spoken audio captions with the semantically relevant portions of natural images that they refer to. We demonstrate that these audio-visual associative localizations emerge from network-internal representations learned as a by-product of training to perform an image-audio retrieval task. Our models operate directly on the image pixels and speech waveform, and do not rely on any conventional supervision in the form of labels, segmentations, or alignments between the modalities during training. We perform analysis using the Places 205 and ADE20k datasets demonstrating that our models implicitly learn semantically-coupled object and word detectors.

Key facts. Cited by 81 papers in the map (data through 14 Oct 2026). First posted on arXiv 4 Apr 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 30; 2022: 22; 2023: 9; 2024: 12; 2025: 4; 2026: 4.

81 papers.

Sort:

By area:

  1. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  2. 2021Localizing Visual Sounds the Hard Waycited by 170
  3. 2022Perceptual Grouping in Contrastive Vision-Language Modelscited by 82
  4. 2021Everything at Once -- Multi-modal Fusion Transformer for Video Retrievalcited by 79
  5. 2022Audio Self-supervised Learning: A Surveycited by 65
  6. 2021Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videoscited by 62
  7. 2022MAViL: Masked Audio-Video Learnerscited by 59
  8. 2021Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completioncited by 58
  9. 2022Mix and Localize: Localizing Sound Sources in Mixturescited by 54
  10. 2021Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptionscited by 53
  11. 2022Contrastive Positive Sample Propagation along the Audio-Visual Event Linecited by 50
  12. 2021Improving Multi-Modal Learning with Uni-Modal Teacherscited by 41
38 more
  1. 2022Word Discovery in Visually Grounded, Self-Supervised Speech Modelscited by 37
  2. 2023Exposing and Mitigating Spurious Correlations for Cross-Modal Retrievalcited by 36
  3. 2021Cross-Modal Discrete Representation Learningcited by 35
  4. 2021Read and Attend: Temporal Localisation in Sign Language Videoscited by 34
  5. 2021The Right to Talk: An Audio-Visual Transformer Approachcited by 33
  6. 2024LeGrad: An Explainability Method for Vision Transformers via Feature Formation Sensitivitycited by 33
  7. 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniquescited by 32
  8. 2021Space-Time Crop & Attend: Improving Cross-modal Video Representation Learningcited by 31
  9. 2021Self-supervised object detection from audio-visual correspondencecited by 29
  10. 2022SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Modelcited by 29
  11. 2021Multimodal Representation Learning via Maximization of Local Mutual Informationcited by 27
  12. 2022Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modelingcited by 25
  13. 2021Fast-Slow Transformer for Visually Grounding Speechcited by 23
  14. 2022Learning Visual Styles from Audio-Visual Associationscited by 22
  15. 2022TVLT: Textless Vision-Language Transformercited by 22
  16. 2021Talk, Don't Write: A Study of Direct Speech-Based Image Retrievalcited by 19
  17. 2021Structure from Silence: Learning Scene Structure from Ambient Soundcited by 17
  18. 2024Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding of Sound and Languagecited by 17
  19. 2025CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignmentcited by 16
  20. 2021Attention-Based Keyword Localisation in Speech using Visual Groundingcited by 13
  21. 2023SD-HuBERT: Sentence-Level Self-Distillation Induces Syllabic Organization in HuBERTcited by 13
  22. 2021Can phones, syllables, and words emerge as side-products of cross-situational audiovisual learning? -- A computational investigationcited by 12
  23. 2022Evaluating context-invariance in unsupervised speech representationscited by 11
  24. 2023Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Modelcited by 11
  25. 2024Cross-Lingual Transfer Learning for Speech Translationcited by 11
  26. 2023What, when, and where? -- Self-Supervised Spatio-Temporal Grounding in Untrimmed Multi-Action Videos from Narrated Instructionscited by 10
  27. 2021Explainable Semantic Space by Grounding Language to Vision with Cross-Modal Contrastive Learningcited by 9
  28. 2021Evaluation of Audio-Visual Alignments in Visually Grounded Speech Modelscited by 8
  29. 2021Cascaded Multilingual Audio-Visual Learning from Videoscited by 8
  30. 2022Self-Supervised Moving Vehicle Detection from Audio-Visual Cuescited by 8
  31. 2022Learning English with Peppa Pigcited by 8
  32. 2022ConceptBeam: Concept Driven Target Speech Extractioncited by 8
  33. 2022YFACC: A Yor`ub'a speech-image dataset for cross-lingual keyword localisation through visual groundingcited by 8
  34. 2024Self-Supervised Audio-Visual Soundscape Stylizationcited by 8
  35. 2022Keyword localisation in untranscribed speech using visually grounded speech modelscited by 7
  36. 2022Using Multiple Instance Learning to Build Multimodal Representationscited by 7
  37. 2021Collaborative Learning to Generate Audio-Video Jointlycited by 6
  38. 2021On the Difficulty of Segmenting Words with Attentioncited by 6

50 most cited of 81 All 81 (JSON)