# Progressive Fusion for Multimodal Integration

arXiv: https://arxiv.org/abs/2209.00302 · HTML: https://mlsanity.com/paper/2209.00302/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 79 references that are in the map.
- First posted on arXiv 1 Sep 2022.

## Builds on (most cited first, 30 of 41)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Measuring Robustness to Natural Distribution Shifts in Image Classification](https://mlsanity.com/paper/2007.00644/) (2020)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [Efficient Low-rank Multimodal Fusion with Modality-Specific Factors](https://mlsanity.com/paper/1806.00064/) (2018)
- [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2005.03545/) (2020)
- [Deep Unfolding Network for Image Super-Resolution](https://mlsanity.com/paper/2003.10428/) (2020)
- [Learning Factorized Multimodal Representations](https://mlsanity.com/paper/1806.06176/) (2018)
- [Memory Fusion Network for Multi-view Sequential Learning](https://mlsanity.com/paper/1802.00927/) (2018)
- [MultiBench: Multiscale Benchmarks for Multimodal Representation Learning](https://mlsanity.com/paper/2107.07502/) (2021)
- [Solving ill-posed inverse problems using iterative deep neural networks](https://mlsanity.com/paper/1704.04058/) (2017)
- [Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2109.00412/) (2021)
- [MMTM: Multimodal Transfer Module for CNN Fusion](https://mlsanity.com/paper/1911.08670/) (2019)
- [Deep Multimodal Fusion by Channel Exchanging](https://mlsanity.com/paper/2011.05005/) (2020)
- [Multi-attention Recurrent Network for Human Communication Comprehension](https://mlsanity.com/paper/1802.00923/) (2018)
- [Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors](https://mlsanity.com/paper/1811.09362/) (2018)
- [Joint Multimodal Learning with Deep Generative Models](https://mlsanity.com/paper/1611.01891/) (2016)
- [CentralNet: a Multilayer Approach for Multimodal Fusion](https://mlsanity.com/paper/1808.07275/) (2018)
- [MFAS: Multimodal Fusion Architecture Search](https://mlsanity.com/paper/1903.06496/) (2019)
- [Deep Unfolding: Model-Based Inspiration of Novel Deep Architectures](https://mlsanity.com/paper/1409.2574/) (2014)
- [Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods](https://mlsanity.com/paper/1907.09358/) (2019)
- [Deep Unfolding for Communications Systems: A Survey and Some New Directions](https://mlsanity.com/paper/1906.05774/) (2019)
- [Contrastive Multimodal Fusion with TupleInfoNCE](https://mlsanity.com/paper/2107.02575/) (2021)
- [Momentum-Net: Fast and convergent iterative neural network for inverse problems](https://mlsanity.com/paper/1907.11818/) (2019)

## Built on by (most cited first, 2 of 2)

- [Multimodal Alignment and Fusion: A Survey](https://mlsanity.com/paper/2411.17040/) (2024)
- [Learning Joint ID-Textual Representation for ID-Preserving Image Synthesis](https://mlsanity.com/paper/2504.14202/) (2025)

## Abstract (arXiv)

Integration of multimodal information from various sources has been shown to boost the performance of machine learning models and thus has received increased attention in recent years. Often such models use deep modality-specific networks to obtain unimodal features which are combined to obtain "late-fusion" representations. However, these designs run the risk of information loss in the respective unimodal pipelines. On the other hand, "early-fusion" methodologies, which combine features early, suffer from the problems associated with feature heterogeneity and high sample complexity. In this work, we present an iterative representation refinement approach, called Progressive Fusion, which mitigates the issues with late fusion representations. Our model-agnostic technique introduces backward connections that make late stage fused representations available to early layers, improving the expressiveness of the representations at those stages, while retaining the advantages of late fusion designs. We test Progressive Fusion on tasks including affective sentiment detection, multimedia analysis, and time series fusion with different models, demonstrating its versatility. We show that our approach consistently improves performance, for instance attaining a 5% reduction in MSE and 40% improvement in robustness on multimodal time series prediction.