mlsanityData to 14 Oct 2026

Temporal and cross-modal attention for audio-visual zero-shot learning

arXiv 2207.09966 ↗First posted 20 Jul 2022cs.CVcited by 19

Not on the map. Why?

Abstract (arXiv)

Audio-visual generalised zero-shot learning for video classification requires understanding the relations between the audio and visual information in order to be able to recognise samples from novel, previously unseen classes at test time. The natural semantic and temporal alignment between audio and visual data in video data can be exploited to learn powerful representations that generalise to unseen classes at test time. We propose a multi-modal and Temporal Cross-attention Framework (modelName) for audio-visual generalised zero-shot learning. Its inputs are temporally aligned audio and visual features that are obtained from pre-trained networks. Encouraging the framework to focus on cross-modal correspondence across time instead of self-attention within the modalities boosts the performance significantly. We show that our proposed framework that ingests temporal features yields state-of-the-art performance on the ucf, vgg, and activity benchmarks for (generalised) zero-shot learning. Code for reproducing all results is available at urlhttps://github.com/ExplainableML/TCAF-GZSL.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 55 of its 78 references that are in the map. First posted on arXiv 20 Jul 2022.

Builds on

55 of 78 references.

References · 55

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  6. 2016Layer Normalizationcited by 5,669
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  10. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  12. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
38 more
  1. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  2. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  3. 2016CNN Architectures for Large-Scale Audio Classificationcited by 704
  4. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  5. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 538
  6. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  7. 2020Multi-modal Transformer for Video Retrievalcited by 395
  8. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  9. 2018Deep Audio-Visual Speech Recognitioncited by 322
  10. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 267
  11. 2017Feature Generating Networks for Zero-Shot Learningcited by 257
  12. 2017Objects that Soundcited by 256
  13. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 249
  14. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 247
  15. 2018X2Face: A network for controlling face generation by using images, audio, and pose codescited by 189
  16. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 188
  17. 2021Localizing Visual Sounds the Hard Waycited by 170
  18. 2014Evaluation of Output Embeddings for Fine-Grained Image Classificationcited by 166
  19. 2015Label-Embedding for Image Classificationcited by 165
  20. 2019f-VAEGAN-D2: A Feature Generating Framework for Any-Shot Learningcited by 164
  21. 2020Attribute Prototype Network for Zero-Shot Learningcited by 160
  22. 2016An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wildcited by 146
  23. 2019Co-Separating Sounds of Visual Objectscited by 110
  24. 2021T2VLAD: Global-Local Sequence Alignment for Text-Video Retrievalcited by 103
  25. 2020Labelling unlabelled videos from scratch with multi-modal self-supervisioncited by 100
  26. 2019ASR is all you need: cross-modal distillation for lip readingcited by 88
  27. 2021HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrievalcited by 87
  28. 2020Foley Music: Learning to Generate Music from Videoscited by 83
  29. 2020Rethinking Zero-shot Video Classification: End-to-end Training for Realistic Applicationscited by 80
  30. 2020A Better Use of Audio-Visual Cues: Dense Video Captioning with Bi-modal Transformercited by 76
  31. 2017Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learningcited by 67
  32. 2021Distilling Audio-Visual Knowledge by Compositional Contrastive Learningcited by 53
  33. 2019Vision-Infused Deep Audio Inpaintingcited by 40
  34. 2019Learning Feature-to-Feature Translator by Alternating Back-Propagation for Generative Zero-Shot Learningcited by 38
  35. 2021Self-supervised object detection from audio-visual correspondencecited by 29
  36. 2019Coordinated Joint Multimodal Embeddings for Generalized Audio-Visual Zeroshot Classification and Retrieval of Videoscited by 27
  37. 2020Large Scale Audiovisual Learning of Sounds with Weakly Labeled Datacited by 27
  38. 2022Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Languagecited by 27

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 6; 2025: 5; 2026: 1.

19 papers.

Sort:

  1. 2023Audio-Visual Class-Incremental Learningcited by 33
  2. 2022iQuery: Instruments as Queries for Audio-Visual Sound Separationcited by 29
  3. 2024Towards Open-Vocabulary Audio-Visual Event Localizationcited by 26
  4. 2023Hyperbolic Audio-visual Zero-shot Learningcited by 19
  5. 2024Spiking Tucker Fusion Transformer for Audio-Visual Zero-Shot Learningcited by 10
  6. 2024Audio-visual Generalized Zero-shot Learning the Easy Waycited by 7
  7. 2025MM-HSD: Multi-Modal Hate Speech Detection in Videoscited by 5
  8. 2023Separating Invisible Sounds Toward Universal Audiovisual Scene-Aware Sound Separationcited by 4
  9. 2024Attend-Fusion: Efficient Audio-Visual Fusion for Video Classificationcited by 3
  10. 2024Multi-label Zero-Shot Audio Classification with Temporal Attentioncited by 2
  11. 2025Adapting to the Unknown: Training-Free Audio-Visual Event Perception with Dynamic Thresholdscited by 2
  12. 2025Multi-Timescale Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learningcited by 2
7 more
  1. 2025Attention-Driven Multimodal Alignment for Long-term Action Quality Assessmentcited by 2
  2. 2023Text-to-feature diffusion for audio-visual few-shot learningcited by 1
  3. 2023Video-adverb retrieval with compositional adverb-action embeddingscited by 1
  4. 2023Boosting Audio-visual Zero-shot Learning with Large Language Modelscited by 1
  5. 2025Extremely Simple Out-of-distribution Detection for Audio-visual Generalized Zero-shot Learningcited by 1
  6. 2024Discrepancy-Aware Attention Network for Enhanced Audio-Visual Zero-Shot Learningcited by 0
  7. 2026CNN-ViT Fusion with Adaptive Attention Gate for Brain Tumor MRI Classification: A Hybrid Deep Learning Modelcited by 0