# Attentional Graph Convolutional Network for Structure-aware Audio-Visual Scene Classification

arXiv: https://arxiv.org/abs/2301.00145 · HTML: https://mlsanity.com/paper/2301.00145/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 81 references that are in the map.
- First posted on arXiv 31 Dec 2022.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Multi-Label Image Recognition with Graph Convolutional Networks](https://mlsanity.com/paper/1904.03582/) (2019)
- [SoundNet: Learning Sound Representations from Unlabeled Video](https://mlsanity.com/paper/1610.09001/) (2016)
- [Deep Convolutional Networks on Graph-Structured Data](https://mlsanity.com/paper/1506.05163/) (2015)
- [A Curated Dataset of Urban Scenes for Audio-Visual Scene Analysis](https://mlsanity.com/paper/2011.00030/) (2020)
- [AttaNet: Attention-Augmented Network for Fast and Accurate Scene Parsing](https://mlsanity.com/paper/2103.05930/) (2021)
- [Places205-VGGNet Models for Scene Recognition](https://mlsanity.com/paper/1508.01667/) (2015)
- [Towards Better Accuracy-efficiency Trade-offs: Divide and Co-training](https://mlsanity.com/paper/2011.14660/) (2020)
- [Object-to-Scene: Learning to Transfer Object Knowledge to Indoor Scene Recognition](https://mlsanity.com/paper/2108.00399/) (2021)
- [ASK: Adaptively Selecting Key Local Features for RGB-D Scene Recognition](https://mlsanity.com/paper/2110.07703/) (2021)
- [DEDUCE: Diverse scEne Detection methods in Unseen Challenging Environments](https://mlsanity.com/paper/1908.00191/) (2019)
- [BORM: Bayesian Object Relation Model for Indoor Scene Recognition](https://mlsanity.com/paper/2108.00397/) (2021)
- [Scene Classification in Indoor Environments for Robots using Context Based Word Embeddings](https://mlsanity.com/paper/1908.06422/) (2019)
- [Depth CNNs for RGB-D scene recognition: learning from scratch better than transferring from RGB-CNNs](https://mlsanity.com/paper/1801.06797/) (2018)
- [Multi-stream Network With Temporal Attention For Environmental Sound Classification](https://mlsanity.com/paper/1901.08608/) (2019)
- [Attention based Convolutional Recurrent Neural Network for Environmental Sound Classification](https://mlsanity.com/paper/1907.02230/) (2019)
- [An Ontology-Aware Framework for Audio Event Classification](https://mlsanity.com/paper/2001.10048/) (2020)
- [Robust Environmental Sound Recognition with Sparse Key-point Encoding and Efficient Multi-spike Learning](https://mlsanity.com/paper/1902.01094/) (2019)
- [Audio-visual scene classification: analysis of DCASE 2021 Challenge submissions](https://mlsanity.com/paper/2105.13675/) (2021)
- [Look and Listen: A Multi-modality Late Fusion Approach to Scene Classification for Autonomous Machines](https://mlsanity.com/paper/2007.10175/) (2020)
- [Feature Pyramid Attention based Residual Neural Network for Environmental Sound Classification](https://mlsanity.com/paper/2205.14411/) (2022)

## Built on by (most cited first, 5 of 5)

- [Inter-object Discriminative Graph Modeling for Indoor Scene Recognition](https://mlsanity.com/paper/2311.05919/) (2023)
- [A Deep Learning-based Global and Segmentation-based Semantic Feature Fusion Approach for Indoor Scene Classification](https://mlsanity.com/paper/2302.06432/) (2023)
- [Semantic-guided modeling of spatial relation and object co-occurrence for indoor scene recognition](https://mlsanity.com/paper/2305.12661/) (2023)
- [Class Relevance Learning For Out-of-distribution Detection](https://mlsanity.com/paper/2401.01021/) (2023)
- [Exploiting Object-based and Segmentation-based Semantic Features for Deep Learning-based Indoor Scene Classification](https://mlsanity.com/paper/2404.07739/) (2024)

## Abstract (arXiv)

Audio-Visual scene understanding is a challenging problem due to the unstructured spatial-temporal relations that exist in the audio signals and spatial layouts of different objects and various texture patterns in the visual images. Recently, many studies have focused on abstracting features from convolutional neural networks while the learning of explicit semantically relevant frames of sound signals and visual images has been overlooked. To this end, we present an end-to-end framework, namely attentional graph convolutional network (AGCN), for structure-aware audio-visual scene representation. First, the spectrogram of sound and input image is processed by a backbone network for feature extraction. Then, to build multi-scale hierarchical information of input features, we utilize an attention fusion mechanism to aggregate features from multiple layers of the backbone network. Notably, to well represent the salient regions and contextual information of audio-visual inputs, the salient acoustic graph (SAG) and contextual acoustic graph (CAG), salient visual graph (SVG), and contextual visual graph (CVG) are constructed for the audio-visual scene representation. Finally, the constructed graphs pass through a graph convolutional network for structure-aware audio-visual scene recognition. Extensive experimental results on the audio, visual and audio-visual scene recognition datasets show that promising results have been achieved by the AGCN methods. Visualizing graphs on the spectrograms and images have been presented to show the effectiveness of proposed CAG/SAG and CVG/SVG that could focus on the salient and semantic relevant regions.