# Hybrid Hypergraph Networks for Multimodal Sequence Data Classification

arXiv: https://arxiv.org/abs/2508.00926 · HTML: https://mlsanity.com/paper/2508.00926/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 34 references that are in the map.
- First posted on arXiv 30 Jul 2025.

## Builds on (most cited first, 17 of 17)

- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting](https://mlsanity.com/paper/1709.04875/) (2017)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Temporal Graph Networks for Deep Learning on Dynamic Graphs](https://mlsanity.com/paper/2006.10637/) (2020)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [TS2Vec: Towards Universal Representation of Time Series](https://mlsanity.com/paper/2106.10466/) (2021)
- [Contrastive Learning of General-Purpose Audio Representations](https://mlsanity.com/paper/2010.10915/) (2020)
- [Trusted Multi-View Classification with Dynamic Evidential Fusion](https://mlsanity.com/paper/2204.11423/) (2022)
- [Self-Supervised Temporal Graph learning with Temporal and Structural Intensity Alignment](https://mlsanity.com/paper/2302.07491/) (2023)
- [Hypergraph Vision Transformers: Images are More than Nodes, More than Edges](https://mlsanity.com/paper/2504.08710/) (2025)
- [Graph Entropy Guided Node Embedding Dimension Selection for Graph Neural Networks](https://mlsanity.com/paper/2105.03178/) (2021)
- [TMac: Temporal Multi-Modal Graph Learning for Acoustic Event Classification](https://mlsanity.com/paper/2309.11845/) (2023)
- [Cross-Modal Learning for Anomaly Detection in Complex Industrial Process: Methodology and Benchmark](https://mlsanity.com/paper/2406.09016/) (2024)
- [Visually-aware Acoustic Event Detection using Heterogeneous Graphs](https://mlsanity.com/paper/2207.07935/) (2022)
- [Multimodal Data Fusion in High-Dimensional Heterogeneous Datasets via Generative Models](https://mlsanity.com/paper/2108.12445/) (2021)
- [Heterogeneous Graph Learning for Acoustic Event Classification](https://mlsanity.com/paper/2303.02665/) (2023)

## Abstract (arXiv)

Modeling temporal multimodal data poses significant challenges in classification tasks, particularly in capturing long-range temporal dependencies and intricate cross-modal interactions. Audiovisual data, as a representative example, is inherently characterized by strict temporal order and diverse modalities. Effectively leveraging the temporal structure is essential for understanding both intra-modal dynamics and inter-modal correlations. However, most existing approaches treat each modality independently and rely on shallow fusion strategies, which overlook temporal dependencies and hinder the model's ability to represent complex structural relationships. To address the limitation, we propose the hybrid hypergraph network (HHN), a novel framework that models temporal multimodal data via a segmentation-first, graph-later strategy. HHN splits sequences into timestamped segments as nodes in a heterogeneous graph. Intra-modal structures are captured via hyperedges guided by a maximum entropy difference criterion, enhancing node heterogeneity and structural discrimination, followed by hypergraph convolution to extract high-order dependencies. Inter-modal links are established through temporal alignment and graph attention for semantic fusion. HHN achieves state-of-the-art (SOTA) results on four multimodal datasets, demonstrating its effectiveness in complex classification tasks.