# MoMA: A Mixture-of-Multimodal-Agents Architecture for Enhancing Clinical Prediction Modelling

arXiv: https://arxiv.org/abs/2508.05492 · HTML: https://mlsanity.com/paper/2508.05492/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 111 references that are in the map.
- First posted on arXiv 7 Aug 2025.

## Builds on (most cited first, 30 of 48)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [MTEB: Massive Text Embedding Benchmark](https://mlsanity.com/paper/2210.07316/) (2022)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [Mixture-of-Agents Enhances Large Language Model Capabilities](https://mlsanity.com/paper/2406.04692/) (2024)
- [MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning](https://mlsanity.com/paper/2311.10537/) (2023)
- [Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks](https://mlsanity.com/paper/2411.04468/) (2024)
- [Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing](https://mlsanity.com/paper/2301.04558/) (2023)
- [OneLLM: One Framework to Align All Modalities with Language](https://mlsanity.com/paper/2312.03700/) (2023)
- [More Agents Is All You Need](https://mlsanity.com/paper/2402.05120/) (2024)
- [The Revolution of Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2402.12451/) (2024)
- [MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models](https://mlsanity.com/paper/2410.13085/) (2024)
- [EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Records](https://mlsanity.com/paper/2401.07128/) (2024)
- [Integrated multimodal artificial intelligence framework for healthcare applications](https://mlsanity.com/paper/2202.12998/) (2022)
- [FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion](https://mlsanity.com/paper/2402.03226/) (2024)
- [XrayGPT: Chest Radiographs Summarization using Medical Vision-Language Models](https://mlsanity.com/paper/2306.07971/) (2023)

## Built on by (most cited first, 7 of 7)

- [Count-Based Approaches Remain Strong: A Benchmark Against Transformer and LLM Pipelines on Structured EHR](https://mlsanity.com/paper/2511.00782/) (2025)
- [Attention-MoA: Enhancing Mixture-of-Agents via Inter-Agent Semantic Attention and Deep Residual Synthesis](https://mlsanity.com/paper/2601.16596/) (2026)
- [Multi-LLM Collaboration for Medication Recommendation](https://mlsanity.com/paper/2512.05066/) (2025)
- [PORTER: Language-Grounded Event Representations for Portable Structured EHR Foundation Models](https://mlsanity.com/paper/2606.24102/) (2026)
- [Traj-Evolve: A Self-Evolving Multi-Agent System for Patient Trajectory Modeling in Lung Cancer Early Detection](https://mlsanity.com/paper/2606.02812/) (2026)
- [Large Language Models as Unified Multimodal Learners for Clinical Prediction](https://mlsanity.com/paper/2607.15380/) (2026)
- [GLoC-EHR: Evidence-Cited Clinical Reasoning over Global Context and Local EHR Events](https://mlsanity.com/paper/2610.01076/) (2026)

## Abstract (arXiv)

Multimodal electronic health record (EHR) data provide richer, complementary insights into patient health compared to single-modality data. However, effectively integrating diverse data modalities for clinical prediction modeling remains challenging due to the substantial data requirements. We introduce a novel architecture, Mixture-of-Multimodal-Agents (MoMA), designed to leverage multiple large language model (LLM) agents for clinical prediction tasks using multimodal EHR data. MoMA employs specialized LLM agents ("specialist agents") to convert non-textual modalities, such as medical images and laboratory results, into structured textual summaries. These summaries, together with clinical notes, are combined by another LLM ("aggregator agent") to generate a unified multimodal summary, which is then used by a third LLM ("predictor agent") to produce clinical predictions. Evaluating MoMA on three prediction tasks using real-world datasets with different modality combinations and prediction settings, MoMA outperforms current state-of-the-art methods, highlighting its enhanced accuracy and flexibility across various tasks.