mlsanityData to 14 Oct 2026

Deep Multimodal Semantic Embeddings for Speech and Images

arXiv 1511.03690 ↗First posted 11 Nov 2015cs.CV cs.AI cs.CLcited by 56

Not on the map. Why?

Abstract (arXiv)

In this paper, we present a model which takes as input a corpus of images with relevant spoken captions and finds a correspondence between the two modalities. We employ a pair of convolutional neural networks to model visual objects and speech signals at the word level, and tie the networks together with an embedding and alignment model which learns a joint semantic space over both modalities. We evaluate our model using image search and annotation tasks on the Flickr8k dataset, which we augmented by collecting a corpus of 40,000 spoken captions using Amazon Mechanical Turk.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Nov 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 15; 2022: 13; 2023: 8; 2024: 13; 2025: 4; 2026: 3.

56 papers.

Sort:

By area:

  1. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  2. 2021Can You Hear It? Backdoor Attacks via Ultrasonic Triggerscited by 44
  3. 2022Word Discovery in Visually Grounded, Self-Supervised Speech Modelscited by 37
  4. 2021Cross-Modal Discrete Representation Learningcited by 35
  5. 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniquescited by 32
  6. 2022SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Modelcited by 29
  7. 2022WAVPROMPT: Towards Few-Shot Spoken Language Understanding with Frozen Language Modelscited by 26
  8. 2022Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modelingcited by 25
  9. 2021Fast-Slow Transformer for Visually Grounding Speechcited by 23
  10. 2022Vision+X: A Survey on Multimodal Learning in the Light of Datacited by 22
  11. 2021Talk, Don't Write: A Study of Direct Speech-Based Image Retrievalcited by 19
  12. 2021Attention-Based Keyword Localisation in Speech using Visual Groundingcited by 13
38 more
  1. 2021Can phones, syllables, and words emerge as side-products of cross-situational audiovisual learning? -- A computational investigationcited by 12
  2. 2022M-SpeechCLIP: Leveraging Large-Scale, Pre-Trained Models for Multilingual Speech to Image Retrievalcited by 9
  3. 2023BIG-C: a Multimodal Multi-Purpose Dataset for Bembacited by 9
  4. 2021#PraCegoVer: A Large Dataset for Image Captioning in Portuguesecited by 8
  5. 2021Timbre Transfer with Variational Auto Encoding and Cycle-Consistent Adversarial Networkscited by 8
  6. 2021Cascaded Multilingual Audio-Visual Learning from Videoscited by 8
  7. 2022Learning English with Peppa Pigcited by 8
  8. 2022YFACC: A Yor`ub'a speech-image dataset for cross-lingual keyword localisation through visual groundingcited by 8
  9. 2022Keyword localisation in untranscribed speech using visually grounded speech modelscited by 7
  10. 2021ZR-2021VG: Zero-Resource Speech Challenge, Visually-Grounded Language Modelling track, 2021 editioncited by 6
  11. 2022Towards visually prompted keyword localisation for zero-resource spoken languagescited by 6
  12. 2021Unsupervised Automatic Speech Recognition: A Reviewcited by 5
  13. 2021Semantic sentence similarity: size does not always mattercited by 5
  14. 2024TMT: Tri-Modal Translation between Speech, Image, and Text by Processing Different Modalities as Different Languagescited by 5
  15. 2026Multi-Way Representation Alignmentcited by 5
  16. 2023Hindi as a Second Language: Improving Visually Grounded Speech with Semantically Similar Samplescited by 4
  17. 2023Visually grounded few-shot word learning in low-resource settingscited by 4
  18. 2023Towards Practical and Efficient Image-to-Speech Captioning with Vision-Language Pre-training and Multi-modal Tokenscited by 4
  19. 2024RU-AI: A Large Multimodal Dataset for Machine-Generated Content Detectioncited by 4
  20. 2024You Only Speak Once to Seecited by 4
  21. 2021Discrete representations in neural models of spoken languagecited by 3
  22. 2022Adversarial Attacks on Speech Recognition Systems for Mission-Critical Applications: A Surveycited by 3
  23. 2024Visually Grounded Speech Models have a Mutual Exclusivity Biascited by 3
  24. 2024Measuring Sound Symbolism in Audio-visual Modelscited by 3
  25. 2021Spoken ObjectNet: A Bias-Controlled Spoken Caption Datasetcited by 2
  26. 2023Fusion-S2iGan: An Efficient and Effective Single-Stage Framework for Speech-to-Image Generationcited by 2
  27. 2023Mitigating Catastrophic Forgetting for Few-Shot Spoken Word Classification Through Meta-Learningcited by 2
  28. 2023Leveraging multilingual transfer for unsupervised semantic acoustic word embeddingscited by 2
  29. 2024Translating speech with just imagescited by 2
  30. 2024VHASR: A Multimodal Speech Recognition System With Vision Hotwordscited by 2
  31. 2023VILAS: Exploring the Effects of Vision and Language Context in Automatic Speech Recognitioncited by 1
  32. 2024Enhancing Audio-Language Models through Self-Supervised Post-Training with Text-Audio Pairscited by 1
  33. 2024Coarse-to-fine Alignment Makes Better Speech-image Retrievalcited by 1
  34. 2025The mutual exclusivity bias of bilingual visually grounded speech modelscited by 1
  35. 2022Modelling word learning and recognition using visually grounded speechcited by 0
  36. 2024Cross-Modal Coordination Across a Diverse Set of Input Modalitiescited by 0
  37. 2024Braille-to-Speech Generator: Audio Generation Based on Joint Fine-Tuning of CLIP and Fastspeech2cited by 0
  38. 2024Cross-Modal Denoising: A Novel Training Paradigm for Enhancing Speech-Image Retrievalcited by 0

50 most cited of 56 All 56 (JSON)