# TMac: Temporal Multi-Modal Graph Learning for Acoustic Event Classification

arXiv: https://arxiv.org/abs/2309.11845 · HTML: https://mlsanity.com/paper/2309.11845/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 66 of its 98 references that are in the map.
- First posted on arXiv 21 Sep 2023.

## Builds on (most cited first, 30 of 66)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Temporal Graph Networks for Deep Learning on Dynamic Graphs](https://mlsanity.com/paper/2006.10637/) (2020)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [Predicting Dynamic Embedding Trajectory in Temporal Interaction Networks](https://mlsanity.com/paper/1908.01207/) (2019)
- [EvolveGCN: Evolving Graph Convolutional Networks for Dynamic Graphs](https://mlsanity.com/paper/1902.10191/) (2019)
- [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021)
- [Self-Supervised MultiModal Versatile Networks](https://mlsanity.com/paper/2006.16228/) (2020)
- [Masked Autoencoders that Listen](https://mlsanity.com/paper/2207.06405/) (2022)
- [Self-Supervised Learning by Cross-Modal Audio-Video Clustering](https://mlsanity.com/paper/1911.12667/) (2019)
- [Objects that Sound](https://mlsanity.com/paper/1712.06651/) (2017)
- [Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization](https://mlsanity.com/paper/1807.00230/) (2018)
- [SoundNet: Learning Sound Representations from Unlabeled Video](https://mlsanity.com/paper/1610.09001/) (2016)
- [Self-supervised Co-training for Video Representation Learning](https://mlsanity.com/paper/2010.09709/) (2020)
- [Efficient Training of Audio Transformers with Patchout](https://mlsanity.com/paper/2110.05069/) (2021)
- [Audio-Visual Instance Discrimination with Cross-Modal Agreement](https://mlsanity.com/paper/2004.12943/) (2020)
- [Contrastive Learning of General-Purpose Audio Representations](https://mlsanity.com/paper/2010.10915/) (2020)
- [AVLnet: Learning Audio-Visual Language Representations from Instructional Videos](https://mlsanity.com/paper/2006.09199/) (2020)
- [Temporal Knowledge Graph Reasoning Based on Evolutional Representation Learning](https://mlsanity.com/paper/2104.10353/) (2021)
- [A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multimodal](https://mlsanity.com/paper/2212.05767/) (2022)
- [Deep Graph Clustering via Dual Correlation Reduction](https://mlsanity.com/paper/2112.14772/) (2021)
- [Evolving Losses for Unsupervised Video Representation Learning](https://mlsanity.com/paper/2002.12177/) (2020)
- [Unsupervised Night Image Enhancement: When Layer Decomposition Meets Light-Effects Suppression](https://mlsanity.com/paper/2207.10564/) (2022)
- [DC-ShadowNet: Single-Image Hard and Soft Shadow Removal Using Unsupervised Domain-Classifier Guided Network](https://mlsanity.com/paper/2207.10434/) (2022)
- [Very Deep Convolutional Neural Networks for Raw Waveforms](https://mlsanity.com/paper/1610.00087/) (2016)

## Built on by (most cited first, 9 of 9)

- [FreCT: Frequency-augmented Convolutional Transformer for Robust Time Series Anomaly Detection](https://mlsanity.com/paper/2505.00941/) (2025)
- [Multi-modal Cross-domain Self-supervised Pre-training for fMRI and EEG Fusion](https://mlsanity.com/paper/2409.19130/) (2024)
- [MLlm-DR: Towards Explainable Depression Recognition with MultiModal Large Language Models](https://mlsanity.com/paper/2507.05591/) (2025)
- [TEn-CATG:Text-Enriched Audio-Visual Video Parsing with Multi-Scale Category-Aware Temporal Graph](https://mlsanity.com/paper/2509.04086/) (2025)
- [Decoupled and Divergence-Conditioned Prompt for Multi-domain Dynamic Graph Foundation Models](https://mlsanity.com/paper/2605.13540/) (2026)
- [A General Benchmark Framework is Dynamic Graph Neural Network Need](https://mlsanity.com/paper/2401.06559/) (2024)
- [Self-Supervised Contrastive Graph Clustering Network via Structural Information Fusion](https://mlsanity.com/paper/2408.04339/) (2024)
- [Hybrid Hypergraph Networks for Multimodal Sequence Data Classification](https://mlsanity.com/paper/2508.00926/) (2025)
- [MRAFnd: Multimodal Retrieval-Augmented Framework for Zero-Shot Fake News Detection](https://mlsanity.com/paper/2608.01430/) (2026)

## Abstract (arXiv)

Audiovisual data is everywhere in this digital age, which raises higher requirements for the deep learning models developed on them. To well handle the information of the multi-modal data is the key to a better audiovisual modal. We observe that these audiovisual data naturally have temporal attributes, such as the time information for each frame in the video. More concretely, such data is inherently multi-modal according to both audio and visual cues, which proceed in a strict chronological order. It indicates that temporal information is important in multi-modal acoustic event modeling for both intra- and inter-modal. However, existing methods deal with each modal feature independently and simply fuse them together, which neglects the mining of temporal relation and thus leads to sub-optimal performance. With this motivation, we propose a Temporal Multi-modal graph learning method for Acoustic event Classification, called TMac, by modeling such temporal information via graph learning techniques. In particular, we construct a temporal graph for each acoustic event, dividing its audio data and video data into multiple segments. Each segment can be considered as a node, and the temporal relationships between nodes can be considered as timestamps on their edges. In this case, we can smoothly capture the dynamic information in intra-modal and inter-modal. Several experiments are conducted to demonstrate TMac outperforms other SOTA models in performance. Our code is available at https://github.com/MGitHubL/TMac.