# Multi-modal Sentiment Analysis using Deep Canonical Correlation Analysis

arXiv: https://arxiv.org/abs/1907.08696 · HTML: https://mlsanity.com/paper/1907.08696/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jul 2019.

## Built on by (most cited first, 7 of 7)

- [Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis](https://mlsanity.com/paper/2107.13669/) (2021)
- [MMLatch: Bottom-up Top-down Fusion for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2201.09828/) (2022)
- [TI-Capsule: Capsule Network for Stock Exchange Prediction](https://mlsanity.com/paper/2102.07718/) (2021)
- [SCoCCA: Multi-modal Sparse Concept Decomposition via Canonical Correlation Analysis](https://mlsanity.com/paper/2603.13884/) (2026)
- [Fusion approaches for emotion recognition from speech using acoustic and text-based features](https://mlsanity.com/paper/2403.18635/) (2024)
- [TCAN: Text-oriented Cross Attention Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2404.04545/) (2024)
- [Enhancing Dialogue Systems with Discourse-Level Understanding Using Deep Canonical Correlation Analysis](https://mlsanity.com/paper/2504.09094/) (2025)

## Abstract (arXiv)

This paper learns multi-modal embeddings from text, audio, and video views/modes of data in order to improve upon down-stream sentiment classification. The experimental framework also allows investigation of the relative contributions of the individual views in the final multi-modal embedding. Individual features derived from the three views are combined into a multi-modal embedding using Deep Canonical Correlation Analysis (DCCA) in two ways i) One-Step DCCA and ii) Two-Step DCCA. This paper learns text embeddings using BERT, the current state-of-the-art in text encoders. We posit that this highly optimized algorithm dominates over the contribution of other views, though each view does contribute to the final result. Classification tasks are carried out on two benchmark datasets and on a new Debate Emotion data set, and together these demonstrate that the one-Step DCCA outperforms the current state-of-the-art in learning multi-modal embeddings.