# CentralNet: a Multilayer Approach for Multimodal Fusion

arXiv: https://arxiv.org/abs/1808.07275 · HTML: https://mlsanity.com/paper/1808.07275/

## Key facts

- Cited by 86 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Aug 2018.

## Built on by (most cited first, 30 of 86)

- [SMIL: Multimodal Learning with Severely Missing Modality](https://mlsanity.com/paper/2103.05677/) (2021)
- [Are Multimodal Transformers Robust to Missing Modality?](https://mlsanity.com/paper/2204.05454/) (2022)
- [Multi-modal Learning with Missing Modality via Shared-Specific Feature Modelling](https://mlsanity.com/paper/2307.14126/) (2023)
- [MultiBench: Multiscale Benchmarks for Multimodal Representation Learning](https://mlsanity.com/paper/2107.07502/) (2021)
- [PMR: Prototypical Modal Rebalance for Multimodal Learning](https://mlsanity.com/paper/2211.07089/) (2022)
- [Deep Multimodal Learning with Missing Modality: A Survey](https://mlsanity.com/paper/2409.07825/) (2024)
- [Boosting Multi-modal Model Performance with Adaptive Gradient Modulation](https://mlsanity.com/paper/2308.07686/) (2023)
- [Dynamic Multimodal Fusion](https://mlsanity.com/paper/2204.00102/) (2022)
- [The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillation](https://mlsanity.com/paper/2206.06487/) (2022)
- [Enhancing multimodal cooperation via sample-level modality valuation](https://mlsanity.com/paper/2309.06255/) (2023)
- [N24News: A New Dataset for Multimodal News Classification](https://mlsanity.com/paper/2108.13327/) (2021)
- [Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions](https://mlsanity.com/paper/2107.13782/) (2021)
- [Classifier-guided Gradient Modulation for Enhanced Multimodal Learning](https://mlsanity.com/paper/2411.01409/) (2024)
- [On-the-fly Modulation for Balanced Multimodal Learning](https://mlsanity.com/paper/2410.11582/) (2024)
- [High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning](https://mlsanity.com/paper/2203.01311/) (2022)
- [Coupled Mamba: Enhanced Multi-modal Fusion with Coupled State Space Model](https://mlsanity.com/paper/2405.18014/) (2024)
- [A Closer Look at Multimodal Representation Collapse](https://mlsanity.com/paper/2505.22483/) (2025)
- [Multi-modal Machine Learning in Engineering Design: A Review and Future Directions](https://mlsanity.com/paper/2302.10909/) (2023)
- [SimMLM: A Simple Framework for Multi-modal Learning with Missing Modality](https://mlsanity.com/paper/2507.19264/) (2025)
- [Revisiting spatio-temporal layouts for compositional action recognition](https://mlsanity.com/paper/2111.01936/) (2021)
- [VaQuitA: Enhancing Alignment in LLM-Assisted Video Understanding](https://mlsanity.com/paper/2312.02310/) (2023)
- [Improving Multimodal Learning with Multi-Loss Gradient Modulation](https://mlsanity.com/paper/2405.07930/) (2024)
- [Learning to Rebalance Multi-Modal Optimization by Adaptively Masking Subnetworks](https://mlsanity.com/paper/2404.08347/) (2024)
- [Balancing Multimodal Training Through Game-Theoretic Regularization](https://mlsanity.com/paper/2411.07335/) (2024)
- [BM-NAS: Bilevel Multimodal Neural Architecture Search](https://mlsanity.com/paper/2104.09379/) (2021)
- [SANCL: Multimodal Review Helpfulness Prediction with Selective Attention and Natural Contrastive Learning](https://mlsanity.com/paper/2209.05040/) (2022)
- [AWSnet: An Auto-weighted Supervision Attention Network for Myocardial Scar and Edema Segmentation in Multi-sequence Cardiac Magnetic Resonance Images](https://mlsanity.com/paper/2201.05344/) (2022)
- [Multimodal Fusion Refiner Networks](https://mlsanity.com/paper/2104.03435/) (2021)
- [Multimodal deep representation learning for quantum cross-platform verification](https://mlsanity.com/paper/2311.03713/) (2023)
- [Harmonic-NAS: Hardware-Aware Multimodal Neural Architecture Search on Resource-constrained Devices](https://mlsanity.com/paper/2309.06612/) (2023)

## Abstract (arXiv)

This paper proposes a novel multimodal fusion approach, aiming to produce best possible decisions by integrating information coming from multiple media. While most of the past multimodal approaches either work by projecting the features of different modalities into the same space, or by coordinating the representations of each modality through the use of constraints, our approach borrows from both visions. More specifically, assuming each modality can be processed by a separated deep convolutional network, allowing to take decisions independently from each modality, we introduce a central network linking the modality specific networks. This central network not only provides a common feature embedding but also regularizes the modality specific networks through the use of multi-task learning. The proposed approach is validated on 4 different computer vision tasks on which it consistently improves the accuracy of existing multimodal fusion approaches.