# Cascaded Multilingual Audio-Visual Learning from Videos

arXiv: https://arxiv.org/abs/2111.04823 · HTML: https://mlsanity.com/paper/2111.04823/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 44 references that are in the map.
- First posted on arXiv 8 Nov 2021.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research](https://mlsanity.com/paper/1904.03493/) (2019)
- [Self-Supervised MultiModal Versatile Networks](https://mlsanity.com/paper/2006.16228/) (2020)
- [How2: A Large-scale Dataset for Multimodal Language Understanding](https://mlsanity.com/paper/1811.00347/) (2018)
- [AVLnet: Learning Audio-Visual Language Representations from Instructional Videos](https://mlsanity.com/paper/2006.09199/) (2020)
- [Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings](https://mlsanity.com/paper/1908.03477/) (2019)
- [Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input](https://mlsanity.com/paper/1804.01452/) (2018)
- [Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos](https://mlsanity.com/paper/2104.12671/) (2021)
- [Deep Multimodal Semantic Embeddings for Speech and Images](https://mlsanity.com/paper/1511.03690/) (2015)
- [Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptions](https://mlsanity.com/paper/2105.04489/) (2021)
- [Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech](https://mlsanity.com/paper/1911.09602/) (2019)
- [Representations of language in a model of visually grounded speech signal](https://mlsanity.com/paper/1702.01991/) (2017)
- [Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modeling](https://mlsanity.com/paper/1810.03459/) (2018)
- [Large-scale representation learning from visually grounded untranscribed speech](https://mlsanity.com/paper/1909.08782/) (2019)
- [Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques](https://mlsanity.com/paper/2104.13225/) (2021)
- [Learning Word-Like Units from Joint Audio-Visual Analysis](https://mlsanity.com/paper/1701.07481/) (2017)
- [Visual Grounding in Video for Unsupervised Word Translation](https://mlsanity.com/paper/2003.05078/) (2020)
- [Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speech](https://mlsanity.com/paper/1804.03052/) (2018)
- [Language learning using Speech to Image retrieval](https://mlsanity.com/paper/1909.03795/) (2019)
- [Visually grounded cross-lingual keyword spotting in speech](https://mlsanity.com/paper/1806.05030/) (2018)
- [Talk, Don't Write: A Study of Direct Speech-Based Image Retrieval](https://mlsanity.com/paper/2104.01894/) (2021)
- [Models of Visually Grounded Speech Signal Pay Attention To Nouns: a Bilingual Experiment on English and Japanese](https://mlsanity.com/paper/1902.03052/) (2019)
- [QuerYD: A video dataset with high-quality text and audio narrations](https://mlsanity.com/paper/2011.11071/) (2020)
- [Learning from Multiview Correlations in Open-Domain Videos](https://mlsanity.com/paper/1811.08890/) (2018)

## Built on by (most cited first, 8 of 8)

- [Towards Video Anomaly Retrieval from Video Anomaly Detection: New Benchmarks and Model](https://mlsanity.com/paper/2307.12545/) (2023)
- [Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languages](https://mlsanity.com/paper/2305.12606/) (2023)
- [M-SpeechCLIP: Leveraging Large-Scale, Pre-Trained Models for Multilingual Speech to Image Retrieval](https://mlsanity.com/paper/2211.01180/) (2022)
- [3MASSIV: Multilingual, Multimodal and Multi-Aspect dataset of Social Media Short Videos](https://mlsanity.com/paper/2203.14456/) (2022)
- [Hindi as a Second Language: Improving Visually Grounded Speech with Semantically Similar Samples](https://mlsanity.com/paper/2303.17517/) (2023)
- [C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video Retrieval](https://mlsanity.com/paper/2210.03625/) (2022)
- [Revitalize Region Feature for Democratizing Video-Language Pre-training of Retrieval](https://mlsanity.com/paper/2203.07720/) (2022)
- [Enhancing Explainability with Multimodal Context Representations for Smarter Robots](https://mlsanity.com/paper/2503.16467/) (2025)

## Abstract (arXiv)

In this paper, we explore self-supervised audio-visual models that learn from instructional videos. Prior work has shown that these models can relate spoken words and sounds to visual content after training on a large-scale dataset of videos, but they were only trained and evaluated on videos in English. To learn multilingual audio-visual representations, we propose a cascaded approach that leverages a model trained on English videos and applies it to audio-visual data in other languages, such as Japanese videos. With our cascaded approach, we show an improvement in retrieval performance of nearly 10x compared to training on the Japanese videos solely. We also apply the model trained on English videos to Japanese and Hindi spoken captions of images, achieving state-of-the-art performance.