# Heterogeneous Graph Learning for Acoustic Event Classification

arXiv: https://arxiv.org/abs/2303.02665 · HTML: https://mlsanity.com/paper/2303.02665/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 5 Mar 2023.

## Builds on (most cited first, 22 of 22)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [Heterogeneous Graph Transformer](https://mlsanity.com/paper/2003.01332/) (2020)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021)
- [MAGNN: Metapath Aggregated Graph Neural Network for Heterogeneous Graph Embedding](https://mlsanity.com/paper/2002.01680/) (2020)
- [Self-Supervised MultiModal Versatile Networks](https://mlsanity.com/paper/2006.16228/) (2020)
- [Are we really making much progress? Revisiting, benchmarking, and refining heterogeneous graph neural networks](https://mlsanity.com/paper/2112.14936/) (2021)
- [Contrastive Learning of General-Purpose Audio Representations](https://mlsanity.com/paper/2010.10915/) (2020)
- [Very Deep Convolutional Neural Networks for Raw Waveforms](https://mlsanity.com/paper/1610.00087/) (2016)
- [Distilling Audio-Visual Knowledge by Compositional Contrastive Learning](https://mlsanity.com/paper/2104.10955/) (2021)
- [A Feature-space Multimodal Data Augmentation Technique for Text-video Retrieval](https://mlsanity.com/paper/2208.02080/) (2022)
- [Learning Speech Representations from Raw Audio by Joint Audiovisual Self-Supervision](https://mlsanity.com/paper/2007.04134/) (2020)
- [Compact Graph Architecture for Speech Emotion Recognition](https://mlsanity.com/paper/2008.02063/) (2020)
- [Self-supervised Graphs for Audio Representation Learning with Limited Labeled Data](https://mlsanity.com/paper/2202.00097/) (2022)
- [Visually-aware Acoustic Event Detection using Heterogeneous Graphs](https://mlsanity.com/paper/2207.07935/) (2022)

## Built on by (most cited first, 2 of 2)

- [TMac: Temporal Multi-Modal Graph Learning for Acoustic Event Classification](https://mlsanity.com/paper/2309.11845/) (2023)
- [Hybrid Hypergraph Networks for Multimodal Sequence Data Classification](https://mlsanity.com/paper/2508.00926/) (2025)

## Abstract (arXiv)

Heterogeneous graphs provide a compact, efficient, and scalable way to model data involving multiple disparate modalities. This makes modeling audiovisual data using heterogeneous graphs an attractive option. However, graph structure does not appear naturally in audiovisual data. Graphs for audiovisual data are constructed manually which is both difficult and sub-optimal. In this work, we address this problem by (i) proposing a parametric graph construction strategy for the intra-modal edges, and (ii) learning the crossmodal edges. To this end, we develop a new model, heterogeneous graph crossmodal network (HGCN) that learns the crossmodal edges. Our proposed model can adapt to various spatial and temporal scales owing to its parametric construction, while the learnable crossmodal edges effectively connect the relevant nodes across modalities. Experiments on a large benchmark dataset (AudioSet) show that our model is state-of-the-art (0.53 mean average precision), outperforming transformer-based models and other graph-based models.