# SFusion: Self-attention based N-to-One Multimodal Fusion Block

arXiv: https://arxiv.org/abs/2208.12776 · HTML: https://mlsanity.com/paper/2208.12776/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 25 references that are in the map.
- First posted on arXiv 26 Aug 2022.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentation](https://mlsanity.com/paper/1809.10486/) (2018)
- [HeMIS: Hetero-Modal Image Segmentation](https://mlsanity.com/paper/1607.05194/) (2016)
- [Hetero-Modal Variational Encoder-Decoder for Joint Modality Completion and Segmentation](https://mlsanity.com/paper/1907.11150/) (2019)
- [Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion](https://mlsanity.com/paper/2002.09708/) (2020)
- [Knowledge distillation from multi-modal to mono-modal segmentation networks](https://mlsanity.com/paper/2106.09564/) (2021)
- [ACN: Adversarial Co-training Network for Brain Tumor Segmentation with Missing Modalities](https://mlsanity.com/paper/2106.14591/) (2021)
- [Latent Correlation Representation Learning for Brain Tumor Segmentation with Missing MRI Modalities](https://mlsanity.com/paper/2104.06231/) (2021)
- [EmbraceNet: A robust deep learning architecture for multimodal classification](https://mlsanity.com/paper/1904.09078/) (2019)
- [Multi-Domain Image Completion for Random Missing Input Data](https://mlsanity.com/paper/2007.05534/) (2020)
- [Representation Disentanglement for Multi-modal brain MR Analysis](https://mlsanity.com/paper/2102.11456/) (2021)
- [A unified representation network for segmentation with missing modalities](https://mlsanity.com/paper/1908.06683/) (2019)
- [EmbraceNet for Activity: A Deep Multimodal Fusion Architecture for Activity Recognition](https://mlsanity.com/paper/2004.13918/) (2020)

## Built on by (most cited first, 14 of 14)

- [Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects](https://mlsanity.com/paper/2311.02332/) (2023)
- [Multimodal Data Integration for Precision Oncology: Challenges and Future Directions](https://mlsanity.com/paper/2406.19611/) (2024)
- [From Screens to Scenes: A Survey of Embodied AI in Healthcare](https://mlsanity.com/paper/2501.07468/) (2025)
- [CogniAlign: Word-Level Multimodal Speech Alignment with Gated Cross-Attention for Alzheimer's Detection](https://mlsanity.com/paper/2506.01890/) (2025)
- [Meta-Learned Modality-Weighted Knowledge Distillation for Robust Multi-Modal Learning with Missing Data](https://mlsanity.com/paper/2405.07155/) (2024)
- [UniMRSeg: Unified Modality-Relax Segmentation via Hierarchical Self-Supervised Compensation](https://mlsanity.com/paper/2509.16170/) (2025)
- [A Multimodal Feature Distillation with Mamba-Transformer Network for Brain Tumor Segmentation with Incomplete Modalities](https://mlsanity.com/paper/2404.14019/) (2024)
- [DeFusion: An Effective Decoupling Fusion Network for Multi-Modal Pregnancy Prediction](https://mlsanity.com/paper/2501.04353/) (2025)
- [Multimodal Knowledge Distillation for Egocentric Action Recognition Robust to Missing Modalities](https://mlsanity.com/paper/2504.08578/) (2025)
- [Complementary Information Mutual Learning for Multimodality Medical Image Segmentation](https://mlsanity.com/paper/2401.02717/) (2024)
- [Robust Brain Tumor Segmentation with Incomplete MRI Modalities Using H"older Divergence and Mutual Information-Enhanced Knowledge Transfer](https://mlsanity.com/paper/2507.01254/) (2025)
- [Early Intervention for VFM-based Multimodal Medical Image Classification](https://mlsanity.com/paper/2603.17514/) (2026)
- [D3Seg: Dependency-Aware Diffusion for Brain Tumor Segmentation with Missing Modalities](https://mlsanity.com/paper/2605.22249/) (2026)
- [LASSNet: Level-Aware Availability-Conditioned Spatial-Semantic Fusion for Brain Tumor Segmentation with Missing MRI Modalities](https://mlsanity.com/paper/2609.06733/) (2026)

## Abstract (arXiv)

People perceive the world with different senses, such as sight, hearing, smell, and touch. Processing and fusing information from multiple modalities enables Artificial Intelligence to understand the world around us more easily. However, when there are missing modalities, the number of available modalities is different in diverse situations, which leads to an N-to-One fusion problem. To solve this problem, we propose a self-attention based fusion block called SFusion. Different from preset formulations or convolution based methods, the proposed block automatically learns to fuse available modalities without synthesizing or zero-padding missing ones. Specifically, the feature representations extracted from upstream processing model are projected as tokens and fed into self-attention module to generate latent multimodal correlations. Then, a modal attention mechanism is introduced to build a shared representation, which can be applied by the downstream decision model. The proposed SFusion can be easily integrated into existing multimodal analysis networks. In this work, we apply SFusion to different backbone networks for human activity recognition and brain tumor segmentation tasks. Extensive experimental results show that the SFusion block achieves better performance than the competing fusion strategies. Our code is available at https://github.com/scut-cszcl/SFusion.