# Graph Capsule Aggregation for Unaligned Multimodal Sequences

arXiv: https://arxiv.org/abs/2108.07543 · HTML: https://mlsanity.com/paper/2108.07543/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 43 references that are in the map.
- First posted on arXiv 17 Aug 2021.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Link Prediction Based on Graph Neural Networks](https://mlsanity.com/paper/1802.09691/) (2018)
- [Hierarchical Graph Representation Learning with Differentiable Pooling](https://mlsanity.com/paper/1806.08804/) (2018)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos](https://mlsanity.com/paper/1606.06259/) (2016)
- [Learning Factorized Multimodal Representations](https://mlsanity.com/paper/1806.06176/) (2018)
- [Memory Fusion Network for Multi-view Sequential Learning](https://mlsanity.com/paper/1802.00927/) (2018)
- [Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities](https://mlsanity.com/paper/1812.07809/) (2018)
- [Multi-attention Recurrent Network for Human Communication Comprehension](https://mlsanity.com/paper/1802.00923/) (2018)
- [Stacked Capsule Autoencoders](https://mlsanity.com/paper/1906.06818/) (2019)
- [Multimodal Language Analysis with Recurrent Multistage Fusion](https://mlsanity.com/paper/1808.03920/) (2018)
- [Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment](https://mlsanity.com/paper/1805.08660/) (2018)
- [Modality-Transferable Emotion Embeddings for Low-Resource Multimodal Emotion Recognition](https://mlsanity.com/paper/2009.09629/) (2020)
- [Factorized Multimodal Transformer for Multimodal Sequential Learning](https://mlsanity.com/paper/1911.09826/) (2019)
- [Analyzing Unaligned Multimodal Sequence via Graph Convolution and Graph Pooling Fusion](https://mlsanity.com/paper/2011.13572/) (2020)
- [Seq2Seq2Sentiment: Multimodal Sequence to Sequence Models for Sentiment Analysis](https://mlsanity.com/paper/1807.03915/) (2018)
- [MTAG: Modal-Temporal Attention Graph for Unaligned Human Multimodal Language Sequences](https://mlsanity.com/paper/2010.11985/) (2020)

## Built on by (most cited first, 6 of 6)

- [Trustworthy Multimodal Fusion for Sentiment Analysis in Ordinal Sentiment Space](https://mlsanity.com/paper/2404.08923/) (2024)
- [Curriculum Learning Meets Weakly Supervised Modality Correlation Learning](https://mlsanity.com/paper/2212.07619/) (2022)
- [Structures Meet Semantics: Multimodal Fusion via Graph Contrastive Learning](https://mlsanity.com/paper/2508.18322/) (2025)
- [Robust Multimodal Sentiment Analysis via Double Information Bottleneck](https://mlsanity.com/paper/2511.01444/) (2025)
- [Mutual Information-based Representations Disentanglement for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/2409.12408/) (2024)
- [MIAR: Modality Interaction and Alignment Representation Fuison for Multimodal Emotion](https://mlsanity.com/paper/2601.02414/) (2026)

## Abstract (arXiv)

Humans express their opinions and emotions through multiple modalities which mainly consist of textual, acoustic and visual modalities. Prior works on multimodal sentiment analysis mostly apply Recurrent Neural Network (RNN) to model aligned multimodal sequences. However, it is unpractical to align multimodal sequences due to different sample rates for different modalities. Moreover, RNN is prone to the issues of gradient vanishing or exploding and it has limited capacity of learning long-range dependency which is the major obstacle to model unaligned multimodal sequences. In this paper, we introduce Graph Capsule Aggregation (GraphCAGE) to model unaligned multimodal sequences with graph-based neural model and Capsule Network. By converting sequence data into graph, the previously mentioned problems of RNN are avoided. In addition, the aggregation capability of Capsule Network and the graph-based structure enable our model to be interpretable and better solve the problem of long-range dependency. Experimental results suggest that GraphCAGE achieves state-of-the-art performance on two benchmark datasets with representations refined by Capsule Network and interpretation provided.