# Hierarchical Semantic-Constrained Heterogeneous Graph for Audio-Visual Event Localization

arXiv: https://arxiv.org/abs/2606.07033 · HTML: https://mlsanity.com/paper/2606.07033/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 43 references that are in the map.
- First posted on arXiv 5 Jun 2026.

## Builds on (most cited first, 18 of 18)

- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [Hyperbolic Entailment Cones for Learning Hierarchical Embeddings](https://mlsanity.com/paper/1804.01882/) (2018)
- [Compositional Entailment Learning for Hyperbolic Vision-Language Models](https://mlsanity.com/paper/2410.06912/) (2024)
- [Positive Sample Propagation along the Audio-Visual Event Line](https://mlsanity.com/paper/2104.00239/) (2021)
- [Inferring Concept Hierarchies from Text Corpora via Hyperbolic Embeddings](https://mlsanity.com/paper/1902.00913/) (2019)
- [MM-Pyramid: Multimodal Pyramid Attentional Network for Audio-Visual Event Localization and Video Parsing](https://mlsanity.com/paper/2111.12374/) (2021)
- [Towards Open-Vocabulary Audio-Visual Event Localization](https://mlsanity.com/paper/2411.11278/) (2024)
- [Hyperbolic Audio-visual Zero-shot Learning](https://mlsanity.com/paper/2308.12558/) (2023)
- [Learning from Untrimmed Videos: Self-Supervised Video Representation Learning with Hierarchical Consistency](https://mlsanity.com/paper/2204.03017/) (2022)
- [Dense Audio-Visual Event Localization under Cross-Modal Consistency and Multi-Temporal Granularity Collaboration](https://mlsanity.com/paper/2412.12628/) (2024)
- [Spiking Tucker Fusion Transformer for Audio-Visual Zero-Shot Learning](https://mlsanity.com/paper/2407.08130/) (2024)
- [Hyperbolic-constraint Point Cloud Reconstruction from Single RGB-D Images](https://mlsanity.com/paper/2412.09055/) (2024)
- [SHMamba: Structured Hyperbolic State Space Model for Audio-Visual Question Answering](https://mlsanity.com/paper/2406.09833/) (2024)
- [Spiking Variational Graph Representation Inference for Video Summarization](https://mlsanity.com/paper/2508.15389/) (2025)
- [Language-Guided Graph Representation Learning for Video Summarization](https://mlsanity.com/paper/2511.10953/) (2025)
- [Multi-Timescale Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learning](https://mlsanity.com/paper/2505.19938/) (2025)

## Built on by (most cited first, 1 of 1)

- [SCoPE: Training-Free Audio-Visual Event Perception via Sparse Cross-Modal Prior Exchange](https://mlsanity.com/paper/2608.07923/) (2026)

## Abstract (arXiv)

Open-vocabulary audio-visual event localization (OV-AVEL) jointly models audio-visual cues to recognize and temporally localize events, including categories unseen during training. Existing methods primarily learn joint audio-visual representations in Euclidean space, but still face two significant challenges. First, the lack of supervision signals for unseen categories makes it difficult to maintain audio-visual consistency across multiple temporal scales. Second, the lack of hierarchical constraints between segment- and video-level semantics prevents the model from establishing semantic consistency across different levels. To address these challenges, we propose a hierarchical semantic constrained heterogeneous graph (HSCHG) for audio-visual event localization framework. We first construct a heterogeneous hierarchical graph in Euclidean space, which includes audio and visual segment nodes and their corresponding video-level nodes. We use multi-directional temporal edges to capture complete temporal information within each modality. Simultaneously, we employ a dual-threshold filtering gated fusion strategy, introducing cross-modal information only when the alignment confidence is high. Furthermore, we introduce bidirectional semantic constraints between segment- and video-level representations to achieve semantic consistency across different levels. Based on this, we map the multi-level audio-visual representations and text prototypes uniformly into hyperbolic space. We use a hierarchical entailment regularization loss to characterize the hierarchical relationships between videos and segments. Extensive experimental results show that our method outperforms existing methods on the OV-AVEL benchmark. Ablation studies further validate the effectiveness of our method.