mlsanityData to 14 Oct 2026

Cascaded Multilingual Audio-Visual Learning from Videos

arXiv 2111.04823 ↗First posted 8 Nov 2021cs.CL cs.CV cs.MMcited by 8

Not on the map. Why?

Abstract (arXiv)

In this paper, we explore self-supervised audio-visual models that learn from instructional videos. Prior work has shown that these models can relate spoken words and sounds to visual content after training on a large-scale dataset of videos, but they were only trained and evaluated on videos in English. To learn multilingual audio-visual representations, we propose a cascaded approach that leverages a model trained on English videos and applies it to audio-visual data in other languages, such as Japanese videos. With our cascaded approach, we show an improvement in retrieval performance of nearly 10x compared to training on the Japanese videos solely. We also apply the model trained on English videos to Japanese and Hindi spoken captions of images, achieving state-of-the-art performance.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 28 of its 44 references that are in the map. First posted on arXiv 8 Nov 2021.

Builds on

28 of 44 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  3. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  4. 2017Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?cited by 551
  5. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
  6. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  7. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  8. 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 138
  9. 2020AVLnet: Learning Audio-Visual Language Representations from Instructional Videoscited by 111
  10. 2019Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddingscited by 101
  11. 2018Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Inputcited by 81
  12. 2021Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videoscited by 62
16 more
  1. 2015Deep Multimodal Semantic Embeddings for Speech and Imagescited by 56
  2. 2021Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptionscited by 53
  3. 2019Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speechcited by 44
  4. 2017Representations of language in a model of visually grounded speech signalcited by 37
  5. 2018Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modelingcited by 34
  6. 2019Large-scale representation learning from visually grounded untranscribed speechcited by 33
  7. 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniquescited by 32
  8. 2017Learning Word-Like Units from Joint Audio-Visual Analysiscited by 27
  9. 2020Visual Grounding in Video for Unsupervised Word Translationcited by 27
  10. 2018Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speechcited by 26
  11. 2019Language learning using Speech to Image retrievalcited by 21
  12. 2018Visually grounded cross-lingual keyword spotting in speechcited by 19
  13. 2021Talk, Don't Write: A Study of Direct Speech-Based Image Retrievalcited by 19
  14. 2019Models of Visually Grounded Speech Signal Pay Attention To Nouns: a Bilingual Experiment on English and Japanesecited by 13
  15. 2020QuerYD: A video dataset with high-quality text and audio narrationscited by 6
  16. 2018Learning from Multiview Correlations in Open-Domain Videoscited by 1

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 3; 2024: 0; 2025: 1.

8 papers.

  1. 2023Towards Video Anomaly Retrieval from Video Anomaly Detection: New Benchmarks and Modelcited by 25
  2. 2023Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languagescited by 20
  3. 2022M-SpeechCLIP: Leveraging Large-Scale, Pre-Trained Models for Multilingual Speech to Image Retrievalcited by 9
  4. 20223MASSIV: Multilingual, Multimodal and Multi-Aspect dataset of Social Media Short Videoscited by 8
  5. 2023Hindi as a Second Language: Improving Visually Grounded Speech with Semantically Similar Samplescited by 4
  6. 2022C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video Retrievalcited by 3
  7. 2022Revitalize Region Feature for Democratizing Video-Language Pre-training of Retrievalcited by 2
  8. 2025Enhancing Explainability with Multimodal Context Representations for Smarter Robotscited by 0