# Low Rank Fusion based Transformers for Multimodal Sequences

arXiv: https://arxiv.org/abs/2007.02038 · HTML: https://mlsanity.com/paper/2007.02038/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jul 2020.

## Built on by (most cited first, 14 of 14)

- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [CubeMLP: An MLP-based Model for Multimodal Sentiment Analysis and Depression Estimation](https://mlsanity.com/paper/2207.14087/) (2022)
- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [A Low-rank Matching Attention based Cross-modal Feature Fusion Method for Conversational Emotion Recognition](https://mlsanity.com/paper/2306.17799/) (2023)
- [Asynchronous Multimodal Video Sequence Fusion via Learning Modality-Exclusive and -Agnostic Representations](https://mlsanity.com/paper/2407.04955/) (2024)
- [Sarcasm in Sight and Sound: Benchmarking and Expansion to Improve Multimodal Sarcasm Detection](https://mlsanity.com/paper/2310.01430/) (2023)
- [Noise-Resistant Multimodal Transformer for Emotion Recognition](https://mlsanity.com/paper/2305.02814/) (2023)
- [DravidianMultiModality: A Dataset for Multi-modal Sentiment Analysis in Tamil and Malayalam](https://mlsanity.com/paper/2106.04853/) (2021)
- [LMR-CBT: Learning Modality-fused Representations with CB-Transformer for Multimodal Emotion Recognition from Unaligned Multimodal Sequences](https://mlsanity.com/paper/2112.01697/) (2021)
- [TACFN: Transformer-based Adaptive Cross-modal Fusion Network for Multimodal Emotion Recognition](https://mlsanity.com/paper/2505.06536/) (2025)
- [Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition](https://mlsanity.com/paper/2305.13583/) (2023)
- [Multi-modal cross-domain mixed fusion model with dual disentanglement for fault diagnosis under unseen working conditions](https://mlsanity.com/paper/2512.24679/) (2025)
- [AdaFuse: Adaptive Multimodal Fusion for Lung Cancer Risk Prediction via Reinforcement Learning](https://mlsanity.com/paper/2602.00347/) (2026)
- [LoRA-Based Cascaded Multimodal Fusion for Action Recognition in Medical Training Environments](https://mlsanity.com/paper/2607.11839/) (2026)

## Abstract (arXiv)

Our senses individually work in a coordinated fashion to express our emotional intentions. In this work, we experiment with modeling modality-specific sensory signals to attend to our latent multimodal emotional intentions and vice versa expressed via low-rank multimodal fusion and multimodal transformers. The low-rank factorization of multimodal fusion amongst the modalities helps represent approximate multiplicative latent signal interactions. Motivated by the work of~citetsai2019MULT and~citeLiu2018, we present our transformer-based cross-fusion architecture without any over-parameterization of the model. The low-rank fusion helps represent the latent signal interactions while the modality-specific attention helps focus on relevant parts of the signal. We present two methods for the Multimodal Sentiment and Emotion Recognition results on CMU-MOSEI, CMU-MOSI, and IEMOCAP datasets and show that our models have lesser parameters, train faster and perform comparably to many larger fusion-based architectures.