# Enhancing Healthcare Recommendation Systems with a Multimodal LLMs-based MOE Architecture

arXiv: https://arxiv.org/abs/2412.11557 · HTML: https://mlsanity.com/paper/2412.11557/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 5 of its 30 references that are in the map.
- First posted on arXiv 16 Dec 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)
- Mixture-of-experts / [Medical and biomedical MoE](https://mlsanity.com/moe/medical-and-biomedical-moe/)

## Builds on (most cited first, 5 of 5)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Automated Scoring of Clinical Patient Notes using Advanced NLP and Pseudo Labeling](https://mlsanity.com/paper/2401.12994/) (2024)
- [Probabilistic Models for Unified Collaborative and Content-Based Recommendation in Sparse-Data Environments](https://mlsanity.com/paper/1301.2303/) (2013)
- [Ensemble BERT: A student social network text sentiment classification model based on ensemble learning and BERT architecture](https://mlsanity.com/paper/2408.04849/) (2024)
- [Enhancing Exploratory Learning through Exploratory Search with the Emergence of Large Language Models](https://mlsanity.com/paper/2408.08894/) (2024)

## Built on by (most cited first, 6 of 6)

- [BadMoE: Backdooring Mixture-of-Experts LLMs via Optimizing Routing Triggers and Infecting Dormant Experts](https://mlsanity.com/paper/2504.18598/) (2025)
- [Intelligent logistics management robot path planning algorithm integrating transformer and GCN network](https://mlsanity.com/paper/2501.02749/) (2025)
- [CPLOYO: A Pulmonary Nodule Detection Model with Multi-Scale Feature Fusion and Nonlinear Feature Learning](https://mlsanity.com/paper/2503.10045/) (2025)
- [Video-VoT-R1: An efficient video inference model integrating image packing and AoE architecture](https://mlsanity.com/paper/2503.15807/) (2025)
- [Hierarchical Multi-Stage BERT Fusion Framework with Dual Attention for Enhanced Cyberbullying Detection in Social Media](https://mlsanity.com/paper/2503.00342/) (2025)
- [FMT:A Multimodal Pneumonia Detection Model Based on Stacking MOE Framework](https://mlsanity.com/paper/2503.05626/) (2025)

## Abstract (arXiv)

With the increasing availability of multimodal data, many fields urgently require advanced architectures capable of effectively integrating these diverse data sources to address specific problems. This study proposes a hybrid recommendation model that combines the Mixture of Experts (MOE) framework with large language models to enhance the performance of recommendation systems in the healthcare domain. We built a small dataset for recommending healthy food based on patient descriptions and evaluated the model's performance on several key metrics, including Precision, Recall, NDCG, and MAP@5. The experimental results show that the hybrid model outperforms the baseline models, which use MOE or large language models individually, in terms of both accuracy and personalized recommendation effectiveness. The paper finds image data provided relatively limited improvement in the performance of the personalized recommendation system, particularly in addressing the cold start problem. Then, the issue of reclassification of images also affected the recommendation results, especially when dealing with low-quality images or changes in the appearance of items, leading to suboptimal performance. The findings provide valuable insights into the development of powerful, scalable, and high-performance recommendation systems, advancing the application of personalized recommendation technologies in real-world domains such as healthcare.