mlsanityData to 7 Oct 2026

Hyperbolic Audio-visual Zero-shot Learning

arXiv 2308.12558 ↗First posted 24 Aug 2023cs.CVcited by 19

Not on the map. Why?

Abstract (arXiv)

Audio-visual zero-shot learning aims to classify samples consisting of a pair of corresponding audio and video sequences from classes that are not present during training. An analysis of the audio-visual data reveals a large degree of hyperbolicity, indicating the potential benefit of using a hyperbolic transformation to achieve curvature-aware geometric learning, with the aim of exploring more complex hierarchical data structures for this task. The proposed approach employs a novel loss function that incorporates cross-modality alignment between video and audio features in the hyperbolic space. Additionally, we explore the use of multiple adaptive curvatures for hyperbolic projections. The experimental results on this very challenging task demonstrate that our proposed hyperbolic approach for zero-shot learning outperforms the SOTA method on three datasets: VGGSound-GZSL, UCF-GZSL, and ActivityNet-GZSL achieving a harmonic mean (HM) improvement of around 3.0%, 7.0%, and 5.3%, respectively.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 39 of its 60 references that are in the map. First posted on arXiv 24 Aug 2023.

Builds on

39 of 60 references.

References · 39

  1. 2017Poincar'e Embeddings for Learning Hierarchical Representationscited by 674
  2. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  3. 2019Similarity-Preserving Knowledge Distillationcited by 507
  4. 2020One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencingcited by 438
  5. 2019Hyperbolic Graph Convolutional Neural Networkscited by 400
  6. 2018Hyperbolic Neural Networkscited by 393
  7. 2018Deep Audio-Visual Speech Recognitioncited by 318
  8. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 246
  9. 2020Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervisioncited by 235
  10. 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
  11. 2019Hyperbolic Image Embeddingscited by 219
  12. 2018Learning to Localize Sound Source in Visual Scenescited by 188
27 more
  1. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 186
  2. 2019Listen to Look: Action Recognition by Previewing Audiocited by 163
  3. 2021Hyperbolic Deep Neural Networks: A Surveycited by 138
  4. 2018Hyperbolic Attention Networkscited by 122
  5. 2020Multiple Sound Sources Localization from Coarse to Finecited by 118
  6. 2022Hyperbolic Image Segmentationcited by 101
  7. 2020Labelling unlabelled videos from scratch with multi-modal self-supervisioncited by 99
  8. 2019Look, Listen, and Act: Towards Audio-Visual Embodied Navigationcited by 95
  9. 2021Robust Audio-Visual Instance Discriminationcited by 80
  10. 2016Ambient Sound Provides Supervision for Visual Learningcited by 77
  11. 2019Hyperbolic Graph Attention Networkcited by 69
  12. 2021Positive Sample Propagation along the Audio-Visual Event Linecited by 69
  13. 2020Discriminative Multi-modality Speech Recognitioncited by 56
  14. 2022Few-Shot Audio-Visual Learning of Environment Acousticscited by 48
  15. 2021HoroPCA: Hyperbolic Dimensionality Reduction via Horospherical Projectionscited by 47
  16. 2020See, Hear, Explore: Curiosity via Audio-Visual Associationcited by 43
  17. 2021CO-SNE: Dimensionality Reduction and Visualization for Hyperbolic Datacited by 28
  18. 2019Coordinated Joint Multimodal Embeddings for Generalized Audio-Visual Zeroshot Classification and Retrieval of Videoscited by 27
  19. 2022Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Languagecited by 27
  20. 2022Learning State-Aware Visual Representations from Audible Interactionscited by 23
  21. 2020AVGZSLNet: Audio-Visual Generalized Zero-Shot Learning by Reconstructing Label Features from Multi-Modal Embeddingscited by 21
  22. 2022Temporal and cross-modal attention for audio-visual zero-shot learningcited by 19
  23. 2022Skin Lesion Recognition with Class-Hierarchy Regularized Hyperbolic Embeddingscited by 19
  24. 2021Adaptive Poincar'e Point to Set Distance for Few-Shot Classificationcited by 15
  25. 2022Curved Geometric Networks for Visual Anomaly Recognitioncited by 15
  26. 2020Self-supervised learning for audio-visual speaker diarizationcited by 14
  27. 2023Hyperbolic Geometry in Computer Vision: A Surveycited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 10; 2026: 2.

19 papers.

Sort:

By area:

  1. 2024Emergent Visual-Semantic Hierarchies in Image-Text Representationscited by 19
  2. 2025Position: Beyond Euclidean -- Foundation Models Should Embrace Non-Euclidean Geometriescited by 15
  3. 2024Spiking Tucker Fusion Transformer for Audio-Visual Zero-Shot Learningcited by 10
  4. 2024FunnyNet-W: Multimodal Learning of Funny Moments in Videos in the Wildcited by 9
  5. 2024SHMamba: Structured Hyperbolic State Space Model for Audio-Visual Question Answeringcited by 7
  6. 2024Audio-visual Generalized Zero-shot Learning the Easy Waycited by 7
  7. 2025Low-distortion and GPU-compatible Tree Embeddings in Hyperbolic Spacecited by 6
  8. 2025Continual Hyperbolic Learning of Instances and Classescited by 4
  9. 2026Path-Decoupled Hyperbolic Flow Matching for Few-Shot Adaptationcited by 4
  10. 2025Curvature Learning for Generalization of Hyperbolic Neural Networkscited by 3
  11. 2025Multi-Timescale Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learningcited by 2
  12. 2023Boosting Audio-visual Zero-shot Learning with Large Language Modelscited by 1
7 more
  1. 2025ESG-Net: Event-Aware Semantic Guided Network for Dense Audio-Visual Event Localizationcited by 1
  2. 2025Zero-Shot Visual Deepfake Detection: Can AI Predict and Prevent Fake Content Before It's Created?cited by 1
  3. 2026Hierarchical Semantic-Constrained Heterogeneous Graph for Audio-Visual Event Localizationcited by 1
  4. 2024Discrepancy-Aware Attention Network for Enhanced Audio-Visual Zero-Shot Learningcited by 0
  5. 2025Extremely Simple Out-of-distribution Detection for Audio-visual Generalized Zero-shot Learningcited by 0
  6. 2025Class Similarity-Based Multimodal Classification under Heterogeneous Category Setscited by 0
  7. 2025More Than A Shortcut: A Hyperbolic Approach To Early-Exit Networkscited by 0