# FMT:A Multimodal Pneumonia Detection Model Based on Stacking MOE Framework

arXiv: https://arxiv.org/abs/2503.05626 · HTML: https://mlsanity.com/paper/2503.05626/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 31 references that are in the map.
- First posted on arXiv 7 Mar 2025.

## Builds on (most cited first, 5 of 5)

- [Wider or Deeper: Revisiting the ResNet Model for Visual Recognition](https://mlsanity.com/paper/1611.10080/) (2016)
- [RobBERT: a Dutch RoBERTa-based Language Model](https://mlsanity.com/paper/2001.06286/) (2020)
- [Enhancing Healthcare Recommendation Systems with a Multimodal LLMs-based MOE Architecture](https://mlsanity.com/paper/2412.11557/) (2024)
- [Ensemble BERT: A student social network text sentiment classification model based on ensemble learning and BERT architecture](https://mlsanity.com/paper/2408.04849/) (2024)
- [Enhancing Exploratory Learning through Exploratory Search with the Emergence of Large Language Models](https://mlsanity.com/paper/2408.08894/) (2024)

## Abstract (arXiv)

Artificial intelligence has shown the potential to improve diagnostic accuracy through medical image analysis for pneumonia diagnosis. However, traditional multimodal approaches often fail to address real-world challenges such as incomplete data and modality loss. In this study, a Flexible Multimodal Transformer (FMT) was proposed, which uses ResNet-50 and BERT for joint representation learning, followed by a dynamic masked attention strategy that simulates clinical modality loss to improve robustness; finally, a sequential mixture of experts (MOE) architecture was used to achieve multi-level decision refinement. After evaluation on a small multimodal pneumonia dataset, FMT achieved state-of-the-art performance with 94% accuracy, 95% recall, and 93% F1 score, outperforming single-modal baselines (ResNet: 89%; BERT: 79%) and the medical benchmark CheXMed (90%), providing a scalable solution for multimodal diagnosis of pneumonia in resource-constrained medical settings.