mlsanityData to 14 Oct 2026

FMT:A Multimodal Pneumonia Detection Model Based on Stacking MOE Framework

arXiv 2503.05626 ↗First posted 7 Mar 2025cs.CV cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Artificial intelligence has shown the potential to improve diagnostic accuracy through medical image analysis for pneumonia diagnosis. However, traditional multimodal approaches often fail to address real-world challenges such as incomplete data and modality loss. In this study, a Flexible Multimodal Transformer (FMT) was proposed, which uses ResNet-50 and BERT for joint representation learning, followed by a dynamic masked attention strategy that simulates clinical modality loss to improve robustness; finally, a sequential mixture of experts (MOE) architecture was used to achieve multi-level decision refinement. After evaluation on a small multimodal pneumonia dataset, FMT achieved state-of-the-art performance with 94% accuracy, 95% recall, and 93% F1 score, outperforming single-modal baselines (ResNet: 89%; BERT: 79%) and the medical benchmark CheXMed (90%), providing a scalable solution for multimodal diagnosis of pneumonia in resource-constrained medical settings.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 5 of its 31 references that are in the map. First posted on arXiv 7 Mar 2025.

Builds on

5 of 31 references.

References · 5

  1. 2016Wider or Deeper: Revisiting the ResNet Model for Visual Recognitioncited by 184
  2. 2020RobBERT: a Dutch RoBERTa-based Language Modelcited by 88
  3. 2024Enhancing Healthcare Recommendation Systems with a Multimodal LLMs-based MOE Architecturecited by 6
  4. 2024Ensemble BERT: A student social network text sentiment classification model based on ensemble learning and BERT architecturecited by 2
  5. 2024Enhancing Exploratory Learning through Exploratory Search with the Emergence of Large Language Modelscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Hierarchical MoE for Multi-Modal ILD Diagnosis
  2. 2024Comprehensive Evaluation of Multimodal AI Models in Medical Imaging Diagnosis: From Data Augmentation to Preference-Based Comparison
  3. 2025Evaluating ChatGPT's Performance in Classifying Pneumonia from Chest X-Ray Images
  4. 2024Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models
  5. 2025MedGemma vs GPT-4: Open-Source and Proprietary Zero-shot Medical Disease Classification from Images
  6. 2025NEURAL: Attention-Guided Pruning for Unified Multimodal Resource-Constrained Clinical Evaluation
  7. 2025MoE-Health: A Mixture of Experts Framework for Robust Multimodal Healthcare Prediction
  8. 2024Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reports