# Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment

arXiv: https://arxiv.org/abs/1805.08660 · HTML: https://mlsanity.com/paper/1805.08660/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 May 2018.

## Built on by (most cited first, 20 of 20)

- [AutoSurvey: Large Language Models Can Automatically Write Surveys](https://mlsanity.com/paper/2406.10252/) (2024)
- [Learning Multi-View Aggregation In the Wild for Large-Scale 3D Semantic Segmentation](https://mlsanity.com/paper/2204.07548/) (2022)
- [Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis](https://mlsanity.com/paper/2107.13669/) (2021)
- [AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models](https://mlsanity.com/paper/2501.16566/) (2025)
- [CubeMLP: An MLP-based Model for Multimodal Sentiment Analysis and Depression Estimation](https://mlsanity.com/paper/2207.14087/) (2022)
- [InterACT: Inter-dependency Aware Action Chunking with Hierarchical Attention Transformers for Bimanual Manipulation](https://mlsanity.com/paper/2409.07914/) (2024)
- [Efficient Multimodal Transformer with Dual-Level Feature Restoration for Robust Multimodal Sentiment Analysis](https://mlsanity.com/paper/2208.07589/) (2022)
- [Masked Graph Learning with Recurrent Alignment for Multimodal Emotion Recognition in Conversation](https://mlsanity.com/paper/2407.16714/) (2024)
- [SpeechFormer++: A Hierarchical Efficient Framework for Paralinguistic Speech Processing](https://mlsanity.com/paper/2302.14638/) (2023)
- [ScaleVLAD: Improving Multimodal Sentiment Analysis via Multi-Scale Fusion of Locally Descriptors](https://mlsanity.com/paper/2112.01368/) (2021)
- [Trustworthy Multimodal Fusion for Sentiment Analysis in Ordinal Sentiment Space](https://mlsanity.com/paper/2404.08923/) (2024)
- [CogniAlign: Word-Level Multimodal Speech Alignment with Gated Cross-Attention for Alzheimer's Detection](https://mlsanity.com/paper/2506.01890/) (2025)
- [Graph Capsule Aggregation for Unaligned Multimodal Sequences](https://mlsanity.com/paper/2108.07543/) (2021)
- [MMLatch: Bottom-up Top-down Fusion for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2201.09828/) (2022)
- [Towards Interpretability in Audio and Visual Affective Machine Learning: A Review](https://mlsanity.com/paper/2306.08933/) (2023)
- [DeepMLF: Multimodal language model with learnable tokens for deep fusion in sentiment analysis](https://mlsanity.com/paper/2504.11082/) (2025)
- [Adapted Multimodal BERT with Layer-wise Fusion for Sentiment Analysis](https://mlsanity.com/paper/2212.00678/) (2022)
- [Complex Mathematical Symbol Definition Structures: A Dataset and Model for Coordination Resolution in Definition Extraction](https://mlsanity.com/paper/2305.14660/) (2023)
- [PowMix: A Versatile Regularizer for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2312.12334/) (2023)
- [Dialog speech sentiment classification for imbalanced datasets](https://mlsanity.com/paper/2109.07228/) (2021)

## Abstract (arXiv)

Multimodal affective computing, learning to recognize and interpret human affects and subjective information from multiple data sources, is still challenging because: (i) it is hard to extract informative features to represent human affects from heterogeneous inputs; (ii) current fusion strategies only fuse different modalities at abstract level, ignoring time-dependent interactions between modalities. Addressing such issues, we introduce a hierarchical multimodal architecture with attention and word-level fusion to classify utter-ance-level sentiment and emotion from text and audio data. Our introduced model outperforms the state-of-the-art approaches on published datasets and we demonstrated that our model is able to visualize and interpret the synchronized attention over modalities.