# Coordinated Joint Multimodal Embeddings for Generalized Audio-Visual Zeroshot Classification and Retrieval of Videos

arXiv: https://arxiv.org/abs/1910.08732 · HTML: https://mlsanity.com/paper/1910.08732/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Oct 2019.

## Built on by (most cited first, 27 of 27)

- [Balanced Multimodal Learning via On-the-fly Gradient Modulation](https://mlsanity.com/paper/2203.15332/) (2022)
- [Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions](https://mlsanity.com/paper/2107.13782/) (2021)
- [On-the-fly Modulation for Balanced Multimodal Learning](https://mlsanity.com/paper/2410.11582/) (2024)
- [Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Language](https://mlsanity.com/paper/2203.03598/) (2022)
- [Towards Open-Vocabulary Audio-Visual Event Localization](https://mlsanity.com/paper/2411.11278/) (2024)
- [Beyond Image to Depth: Improving Depth Prediction using Echoes](https://mlsanity.com/paper/2103.08468/) (2021)
- [Multimodal Open-Vocabulary Video Classification via Pre-Trained Vision and Language Models](https://mlsanity.com/paper/2207.07646/) (2022)
- [Temporal and cross-modal attention for audio-visual zero-shot learning](https://mlsanity.com/paper/2207.09966/) (2022)
- [Hyperbolic Audio-visual Zero-shot Learning](https://mlsanity.com/paper/2308.12558/) (2023)
- [Beyond Mono to Binaural: Generating Binaural Audio from Mono Audio with Depth and Cross Modal Attention](https://mlsanity.com/paper/2111.08046/) (2021)
- [Multimodal Imbalance-Aware Gradient Modulation for Weakly-supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2307.02041/) (2023)
- [Spiking Tucker Fusion Transformer for Audio-Visual Zero-Shot Learning](https://mlsanity.com/paper/2407.08130/) (2024)
- [CrossATNet - A Novel Cross-Attention Based Framework for Sketch-Based Image Retrieval](https://mlsanity.com/paper/2104.09918/) (2021)
- [Lift-Attend-Splat: Bird's-eye-view camera-lidar fusion using transformers](https://mlsanity.com/paper/2312.14919/) (2023)
- [Audio-visual Generalized Zero-shot Learning the Easy Way](https://mlsanity.com/paper/2407.13095/) (2024)
- [Learning Speaker-specific Lip-to-Speech Generation](https://mlsanity.com/paper/2206.02050/) (2022)
- [Adapting to the Unknown: Training-Free Audio-Visual Event Perception with Dynamic Thresholds](https://mlsanity.com/paper/2503.13693/) (2025)
- [Multi-Timescale Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learning](https://mlsanity.com/paper/2505.19938/) (2025)
- [Discriminative Semantic Transitive Consistency for Cross-Modal Learning](https://mlsanity.com/paper/2103.14103/) (2021)
- [Boosting Audio-visual Zero-shot Learning with Large Language Models](https://mlsanity.com/paper/2311.12268/) (2023)
- [DETECLAP: Enhancing Audio-Visual Representation Learning with Object Information](https://mlsanity.com/paper/2409.11729/) (2024)
- [Extremely Simple Out-of-distribution Detection for Audio-visual Generalized Zero-shot Learning](https://mlsanity.com/paper/2503.22197/) (2025)
- [Zero-Shot Visual Deepfake Detection: Can AI Predict and Prevent Fake Content Before It's Created?](https://mlsanity.com/paper/2509.18461/) (2025)
- [HAVE-Net: Hallucinated Audio-Visual Embeddings for Few-Shot Classification with Unimodal Cues](https://mlsanity.com/paper/2309.13470/) (2023)
- [Discrepancy-Aware Attention Network for Enhanced Audio-Visual Zero-Shot Learning](https://mlsanity.com/paper/2412.11715/) (2024)
- [MPJudge: Towards Perceptual Assessment of Music-Induced Paintings](https://mlsanity.com/paper/2511.07137/) (2025)
- [RA-SSU: Towards Fine-Grained Audio-Visual Learning with Region-Aware Sound Source Understanding](https://mlsanity.com/paper/2603.09809/) (2026)

## Abstract (arXiv)

We present an audio-visual multimodal approach for the task of zeroshot learning (ZSL) for classification and retrieval of videos. ZSL has been studied extensively in the recent past but has primarily been limited to visual modality and to images. We demonstrate that both audio and visual modalities are important for ZSL for videos. Since a dataset to study the task is currently not available, we also construct an appropriate multimodal dataset with 33 classes containing 156,416 videos, from an existing large scale audio event dataset. We empirically show that the performance improves by adding audio modality for both tasks of zeroshot classification and retrieval, when using multimodal extensions of embedding learning methods. We also propose a novel method to predict the `dominant' modality using a jointly learned modality attention network. We learn the attention in a semi-supervised setting and thus do not require any additional explicit labelling for the modalities. We provide qualitative validation of the modality specific attention, which also successfully generalizes to unseen test classes.