# Deep Multimodal Fusion of Data with Heterogeneous Dimensionality via Projective Networks

arXiv: https://arxiv.org/abs/2402.01311 · HTML: https://mlsanity.com/paper/2402.01311/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 60 references that are in the map.
- First posted on arXiv 2 Feb 2024.

## Builds on (most cited first, 19 of 19)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations](https://mlsanity.com/paper/1707.03237/) (2017)
- [ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation](https://mlsanity.com/paper/1606.02147/) (2016)
- [REFUGE Challenge: A Unified Framework for Evaluating Automated Methods for Glaucoma Assessment from Fundus Photographs](https://mlsanity.com/paper/1910.03667/) (2019)
- [Unified Focal loss: Generalising Dice and cross entropy-based losses to handle class imbalanced medical image segmentation](https://mlsanity.com/paper/2102.04525/) (2021)
- [A review: Deep learning for medical image segmentation using multi-modality fusion](https://mlsanity.com/paper/2004.10664/) (2020)
- [Multispectral Deep Neural Networks for Pedestrian Detection](https://mlsanity.com/paper/1611.02644/) (2016)
- [Superhuman Accuracy on the SNEMI3D Connectomics Challenge](https://mlsanity.com/paper/1706.00120/) (2017)
- [Ensembles of Multiple Models and Architectures for Robust Brain Tumour Segmentation](https://mlsanity.com/paper/1711.01468/) (2017)
- [OCTA-500: A Retinal Dataset for Optical Coherence Tomography Angiography Study](https://mlsanity.com/paper/2012.07261/) (2020)
- [GAMMA Challenge:Glaucoma grAding from Multi-Modality imAges](https://mlsanity.com/paper/2202.06511/) (2022)
- [IVD-Net: Intervertebral disc localization and segmentation in MRI with a multi-modal UNet](https://mlsanity.com/paper/1811.08305/) (2018)
- [Projective Skip-Connections for Segmentation Along a Subset of Dimensions in Retinal OCT](https://mlsanity.com/paper/2108.00831/) (2021)
- [Simultaneous segmentation and classification of the retinal arteries and veins from color fundus images](https://mlsanity.com/paper/2209.09582/) (2022)
- [Segmentation of Bruch's Membrane in retinal OCT with AMD using anatomical priors and uncertainty quantification](https://mlsanity.com/paper/2210.14799/) (2022)
- [Self-supervised learning via inter-modal reconstruction and feature projection networks for label-efficient 3D-to-2D segmentation](https://mlsanity.com/paper/2307.03008/) (2023)
- [Liver segmentation in CT images using three dimensional to two dimensional fully convolutional network](https://mlsanity.com/paper/1802.07800/) (2018)

## Built on by (most cited first, 3 of 3)

- [Masked Image Modelling for retinal OCT understanding](https://mlsanity.com/paper/2405.14788/) (2024)
- [PiPViT: Patch-based Visual Interpretable Prototypes for Retinal Image Analysis](https://mlsanity.com/paper/2506.10669/) (2025)
- [Fully Automated High-Precision Segmentation of Retinal Atrophy and Ellipsoid Zone Thickness in OCT: A Reliable Tool for Real-World GA Monitoring](https://mlsanity.com/paper/2606.31502/) (2026)

## Abstract (arXiv)

The use of multimodal imaging has led to significant improvements in the diagnosis and treatment of many diseases. Similar to clinical practice, some works have demonstrated the benefits of multimodal fusion for automatic segmentation and classification using deep learning-based methods. However, current segmentation methods are limited to fusion of modalities with the same dimensionality (e.g., 3D+3D, 2D+2D), which is not always possible, and the fusion strategies implemented by classification methods are incompatible with localization tasks. In this work, we propose a novel deep learning-based framework for the fusion of multimodal data with heterogeneous dimensionality (e.g., 3D+2D) that is compatible with localization tasks. The proposed framework extracts the features of the different modalities and projects them into the common feature subspace. The projected features are then fused and further processed to obtain the final prediction. The framework was validated on the following tasks: segmentation of geographic atrophy (GA), a late-stage manifestation of age-related macular degeneration, and segmentation of retinal blood vessels (RBV) in multimodal retinal imaging. Our results show that the proposed method outperforms the state-of-the-art monomodal methods on GA and RBV segmentation by up to 3.10% and 4.64% Dice, respectively.