# AVGZSLNet: Audio-Visual Generalized Zero-Shot Learning by Reconstructing Label Features from Multi-Modal Embeddings

arXiv: https://arxiv.org/abs/2005.13402 · HTML: https://mlsanity.com/paper/2005.13402/

## Key facts

- Cited by 21 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 27 May 2020.

## Built on by (most cited first, 21 of 21)

- [Sound-Guided Semantic Image Manipulation](https://mlsanity.com/paper/2112.00007/) (2021)
- [Audio-Visual Class-Incremental Learning](https://mlsanity.com/paper/2308.11073/) (2023)
- [Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Language](https://mlsanity.com/paper/2203.03598/) (2022)
- [Towards Open-Vocabulary Audio-Visual Event Localization](https://mlsanity.com/paper/2411.11278/) (2024)
- [Beyond Image to Depth: Improving Depth Prediction using Echoes](https://mlsanity.com/paper/2103.08468/) (2021)
- [Multimodal Open-Vocabulary Video Classification via Pre-Trained Vision and Language Models](https://mlsanity.com/paper/2207.07646/) (2022)
- [Temporal and cross-modal attention for audio-visual zero-shot learning](https://mlsanity.com/paper/2207.09966/) (2022)
- [Hyperbolic Audio-visual Zero-shot Learning](https://mlsanity.com/paper/2308.12558/) (2023)
- [Beyond Mono to Binaural: Generating Binaural Audio from Mono Audio with Depth and Cross Modal Attention](https://mlsanity.com/paper/2111.08046/) (2021)
- [Spiking Tucker Fusion Transformer for Audio-Visual Zero-Shot Learning](https://mlsanity.com/paper/2407.08130/) (2024)
- [Robust Sound-Guided Image Manipulation](https://mlsanity.com/paper/2208.14114/) (2022)
- [Audio-visual Generalized Zero-shot Learning the Easy Way](https://mlsanity.com/paper/2407.13095/) (2024)
- [REIS: A High-Performance and Energy-Efficient Retrieval System with In-Storage Processing](https://mlsanity.com/paper/2506.16444/) (2025)
- [Adapting to the Unknown: Training-Free Audio-Visual Event Perception with Dynamic Thresholds](https://mlsanity.com/paper/2503.13693/) (2025)
- [Multi-Timescale Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learning](https://mlsanity.com/paper/2505.19938/) (2025)
- [Discriminative Semantic Transitive Consistency for Cross-Modal Learning](https://mlsanity.com/paper/2103.14103/) (2021)
- [Boosting Audio-visual Zero-shot Learning with Large Language Models](https://mlsanity.com/paper/2311.12268/) (2023)
- [Discrepancy-Aware Attention Network for Enhanced Audio-Visual Zero-Shot Learning](https://mlsanity.com/paper/2412.11715/) (2024)
- [Extremely Simple Out-of-distribution Detection for Audio-visual Generalized Zero-shot Learning](https://mlsanity.com/paper/2503.22197/) (2025)
- [Class Similarity-Based Multimodal Classification under Heterogeneous Category Sets](https://mlsanity.com/paper/2506.09745/) (2025)
- [MPJudge: Towards Perceptual Assessment of Music-Induced Paintings](https://mlsanity.com/paper/2511.07137/) (2025)

## Abstract (arXiv)

In this paper, we propose a novel approach for generalized zero-shot learning in a multi-modal setting, where we have novel classes of audio/video during testing that are not seen during training. We use the semantic relatedness of text embeddings as a means for zero-shot learning by aligning audio and video embeddings with the corresponding class label text feature space. Our approach uses a cross-modal decoder and a composite triplet loss. The cross-modal decoder enforces a constraint that the class label text features can be reconstructed from the audio and video embeddings of data points. This helps the audio and video embeddings to move closer to the class label text embedding. The composite triplet loss makes use of the audio, video, and text embeddings. It helps bring the embeddings from the same class closer and push away the embeddings from different classes in a multi-modal setting. This helps the network to perform better on the multi-modal zero-shot learning task. Importantly, our multi-modal zero-shot learning approach works even if a modality is missing at test time. We test our approach on the generalized zero-shot classification and retrieval tasks and show that our approach outperforms other models in the presence of a single modality as well as in the presence of multiple modalities. We validate our approach by comparing it with previous approaches and using various ablations.