mlsanityData to 14 Oct 2026

Hindi as a Second Language: Improving Visually Grounded Speech with Semantically Similar Samples

arXiv 2303.17517 ↗First posted 30 Mar 2023cs.CL cs.CV cs.SDcited by 4

Not on the map. Why?

Abstract (arXiv)

The objective of this work is to explore the learning of visually grounded speech models (VGS) from multilingual perspective. Bilingual VGS models are generally trained with an equal number of spoken captions from both languages. However, in reality, there can be an imbalance among the languages for the available spoken captions. Our key contribution in this work is to leverage the power of a high-resource language in a bilingual visually grounded speech model to improve the performance of a low-resource language. We introduce two methods to distill the knowledge of high-resource language into low-resource languages: (1) incorporating a strong pre-trained high-resource language encoder and (2) using semantically similar spoken captions. Our experiments show that combining these two approaches effectively enables the low-resource language to surpass the performances of monolingual and bilingual counterparts for cross-modal retrieval tasks.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 22 of its 29 references that are in the map. First posted on arXiv 30 Mar 2023.

Builds on

22 of 29 references.

References · 22

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  2. 2020Supervised Contrastive Learningcited by 2,829
  3. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  4. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 326
  5. 2020Self-supervised Co-training for Video Representation Learningcited by 232
  6. 2020AVLnet: Learning Audio-Visual Language Representations from Instructional Videoscited by 111
  7. 2018Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Inputcited by 81
  8. 2015Deep Multimodal Semantic Embeddings for Speech and Imagescited by 56
  9. 2019Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speechcited by 44
  10. 2017Representations of language in a model of visually grounded speech signalcited by 37
  11. 2022Word Discovery in Visually Grounded, Self-Supervised Speech Modelscited by 37
  12. 2022Learning Sound Localization Better From Semantically Similar Samplescited by 32
10 more
  1. 2017Learning Word-Like Units from Joint Audio-Visual Analysiscited by 27
  2. 2018Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speechcited by 26
  3. 2018Visually grounded cross-lingual keyword spotting in speechcited by 19
  4. 2017Visually grounded learning of keyword prediction from untranscribed speechcited by 16
  5. 2021Attention-Based Keyword Localisation in Speech using Visual Groundingcited by 13
  6. 2019Semantic query-by-example speech search using visual groundingcited by 11
  7. 2018Symbolic inductive bias for visually grounded learning of spoken languagecited by 10
  8. 2020Textual Supervision for Visually Grounded Spoken Language Understandingcited by 9
  9. 2021Cascaded Multilingual Audio-Visual Learning from Videoscited by 8
  10. 2020Globetrotter: Connecting Languages by Connecting Imagescited by 6

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3.

4 papers.

  1. 2023Sound Source Localization is All about Cross-Modal Alignmentcited by 31
  2. 2024Aligning Sight and Sound: Advanced Sound Source Localization Through Audio-Visual Alignmentcited by 8
  3. 2024Let Me Finish My Sentence: Video Temporal Grounding with Holistic Text Understandingcited by 7
  4. 2024ElectroVizQA: How well do Multi-modal LLMs perform in Electronics Visual Question Answering?cited by 4