# Memory Fusion Network for Multi-view Sequential Learning

arXiv: https://arxiv.org/abs/1802.00927 · HTML: https://mlsanity.com/paper/1802.00927/

## Key facts

- Cited by 168 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Feb 2018.

## Built on by (most cited first, 30 of 168)

- [Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2102.04830/) (2021)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation](https://mlsanity.com/paper/2107.06779/) (2021)
- [Decoupled Multimodal Distilling for Emotion Recognition](https://mlsanity.com/paper/2303.13802/) (2023)
- [M3Care: Learning with Missing Modalities in Multimodal Healthcare Data](https://mlsanity.com/paper/2210.17292/) (2022)
- [MM-DFN: Multimodal Dynamic Fusion Network for Emotion Recognition in Conversations](https://mlsanity.com/paper/2203.02385/) (2022)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Transformer-based Multimodal Information Fusion for Facial Expression Analysis](https://mlsanity.com/paper/2203.12367/) (2022)
- [Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations](https://mlsanity.com/paper/2210.17444/) (2022)
- [A Review on Explainability in Multimodal Deep Neural Nets](https://mlsanity.com/paper/2105.07878/) (2021)
- [Learning in Audio-visual Context: A Review, Analysis, and New Perspective](https://mlsanity.com/paper/2208.09579/) (2022)
- [Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Module](https://mlsanity.com/paper/2209.02604/) (2022)
- [Hybrid Contrastive Learning of Tri-Modal Representation for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2109.01797/) (2021)
- [MIntRec: A New Dataset for Multimodal Intent Recognition](https://mlsanity.com/paper/2209.04355/) (2022)
- [Multimodal End-to-End Sparse Model for Emotion Recognition](https://mlsanity.com/paper/2103.09666/) (2021)
- [CubeMLP: An MLP-based Model for Multimodal Sentiment Analysis and Depression Estimation](https://mlsanity.com/paper/2207.14087/) (2022)
- [GPT-4V with Emotion: A Zero-shot Benchmark for Generalized Emotion Recognition](https://mlsanity.com/paper/2312.04293/) (2023)
- [Multimodal Learning using Optimal Transport for Sarcasm and Humor Detection](https://mlsanity.com/paper/2110.10949/) (2021)
- [A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations](https://mlsanity.com/paper/2310.20494/) (2023)
- [Multi-modal Sarcasm Detection and Humor Classification in Code-mixed Conversations](https://mlsanity.com/paper/2105.09984/) (2021)
- [A Comprehensive Survey on Multi-modal Conversational Emotion Recognition with Deep Learning](https://mlsanity.com/paper/2312.05735/) (2023)
- [Memory-Guided Multi-View Multi-Domain Fake News Detection](https://mlsanity.com/paper/2206.12808/) (2022)
- [Coupled Mamba: Enhanced Multi-modal Fusion with Coupled State Space Model](https://mlsanity.com/paper/2405.18014/) (2024)
- [M2Lens: Visualizing and Explaining Multimodal Models for Sentiment Analysis](https://mlsanity.com/paper/2107.08264/) (2021)
- [COGMEN: COntextualized GNN based Multimodal Emotion recognitioN](https://mlsanity.com/paper/2205.02455/) (2022)
- [A Low-rank Matching Attention based Cross-modal Feature Fusion Method for Conversational Emotion Recognition](https://mlsanity.com/paper/2306.17799/) (2023)
- [Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation](https://mlsanity.com/paper/2310.03986/) (2023)
- [DLF: Disentangled-Language-Focused Multimodal Sentiment Analysis](https://mlsanity.com/paper/2412.12225/) (2024)
- [MIntRec2.0: A Large-scale Benchmark Dataset for Multimodal Intent Recognition and Out-of-scope Detection in Conversations](https://mlsanity.com/paper/2403.10943/) (2024)
- [Efficient Long-distance Latent Relation-aware Graph Neural Network for Multi-modal Emotion Recognition in Conversations](https://mlsanity.com/paper/2407.00119/) (2024)

## Abstract (arXiv)

Multi-view sequential learning is a fundamental problem in machine learning dealing with multi-view sequences. In a multi-view sequence, there exists two forms of interactions between different views: view-specific interactions and cross-view interactions. In this paper, we present a new neural architecture for multi-view sequential learning called the Memory Fusion Network (MFN) that explicitly accounts for both interactions in a neural architecture and continuously models them through time. The first component of the MFN is called the System of LSTMs, where view-specific interactions are learned in isolation through assigning an LSTM function to each view. The cross-view interactions are then identified using a special attention mechanism called the Delta-memory Attention Network (DMAN) and summarized through time with a Multi-view Gated Memory. Through extensive experimentation, MFN is compared to various proposed approaches for multi-view sequential learning on multiple publicly available benchmark datasets. MFN outperforms all the existing multi-view approaches. Furthermore, MFN outperforms all current state-of-the-art models, setting new state-of-the-art results for these multi-view datasets.