# BiMediX: Bilingual Medical Mixture of Experts LLM

arXiv: https://arxiv.org/abs/2402.13253 · HTML: https://mlsanity.com/paper/2402.13253/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 50 references that are in the map.
- First posted on arXiv 20 Feb 2024.

## Builds on (most cited first, 27 of 27)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data](https://mlsanity.com/paper/2304.08247/) (2023)
- [LinkBERT: Pretraining Language Models with Document Links](https://mlsanity.com/paper/2203.15827/) (2022)
- [Deep Bidirectional Language-Knowledge Graph Pretraining](https://mlsanity.com/paper/2210.09338/) (2022)
- [BioMegatron: Larger Biomedical Domain Language Model](https://mlsanity.com/paper/2010.06060/) (2020)
- [AraT5: Text-to-Text Transformers for Arabic Language Generation](https://mlsanity.com/paper/2109.12068/) (2021)
- [AraBART: a Pretrained Arabic Sequence-to-Sequence Model for Abstractive Summarization](https://mlsanity.com/paper/2203.10945/) (2022)

## Built on by (most cited first, 21 of 21)

- [LLM Post-Training: A Deep Dive into Reasoning Large Language Models](https://mlsanity.com/paper/2502.21321/) (2025)
- [A Review of Large Language Models and Autonomous Agents in Chemistry](https://mlsanity.com/paper/2407.01603/) (2024)
- [Vision-Language Models for Edge Networks: A Comprehensive Survey](https://mlsanity.com/paper/2502.07855/) (2025)
- [Ask Patients with Patience: Enabling LLMs for Human-Centric Medical Dialogue with Grounded Reasoning](https://mlsanity.com/paper/2502.07143/) (2025)
- [Large Language Model Benchmarks in Medical Tasks](https://mlsanity.com/paper/2410.21348/) (2024)
- [A Survey on Large Language Models from General Purpose to Medical Applications: Datasets, Methodologies, and Evaluations](https://mlsanity.com/paper/2406.10303/) (2024)
- [Mixture of Experts in Large Language Models](https://mlsanity.com/paper/2507.11181/) (2025)
- [MediX-R1: Open Ended Medical Reinforcement Learning](https://mlsanity.com/paper/2602.23363/) (2026)
- [MoPEFT: A Mixture-of-PEFTs for the Segment Anything Model](https://mlsanity.com/paper/2405.00293/) (2024)
- [Development and bilingual evaluation of Japanese medical large language model within reasonably low computational resources](https://mlsanity.com/paper/2409.11783/) (2024)
- [A Comprehensive Evaluation of Large Language Models on Mental Illnesses in Arabic Context](https://mlsanity.com/paper/2501.06859/) (2025)
- [MedAraBench: Large-Scale Arabic Medical Question Answering Dataset and Benchmark](https://mlsanity.com/paper/2602.01714/) (2026)
- [GemmAr: Enhancing LLMs Through Arabic Instruction-Tuning](https://mlsanity.com/paper/2407.02147/) (2024)
- [Reviewing Clinical Knowledge in Medical Large Language Models: Training and Beyond](https://mlsanity.com/paper/2502.20988/) (2025)
- [MedAidDialog: A Multilingual Multi-Turn Medical Dialogue Dataset for Accessible Healthcare](https://mlsanity.com/paper/2603.24132/) (2026)
- [OMIND: Framework for Knowledge Grounded Finetuning and Multi-Turn Dialogue Benchmark for Mental Health LLMs](https://mlsanity.com/paper/2603.25105/) (2026)
- [Towards Democratizing Multilingual Large Language Models For Medicine Through A Two-Stage Instruction Fine-tuning Approach](https://mlsanity.com/paper/2409.05732/) (2024)
- [PerMedCQA: Benchmarking Large Language Models on Medical Consumer Question Answering in Persian Language](https://mlsanity.com/paper/2505.18331/) (2025)
- [IndicMedDialog: A Parallel Multi-Turn Medical Dialogue Dataset for Accessible Healthcare in Indic Languages](https://mlsanity.com/paper/2605.13292/) (2026)
- [Trade-offs in Medical LLM Adaptation: An Empirical Study in French QA](https://mlsanity.com/paper/2606.19266/) (2026)
- [Bridging the English-Arabic Medical Knowledge Gap: Targeted Low-Rank Adaptation via Causal Layer Selection](https://mlsanity.com/paper/2608.00207/) (2026)

## Abstract (arXiv)

In this paper, we introduce BiMediX, the first bilingual medical mixture of experts LLM designed for seamless interaction in both English and Arabic. Our model facilitates a wide range of medical interactions in English and Arabic, including multi-turn chats to inquire about additional details such as patient symptoms and medical history, multiple-choice question answering, and open-ended question answering. We propose a semi-automated English-to-Arabic translation pipeline with human refinement to ensure high-quality translations. We also introduce a comprehensive evaluation benchmark for Arabic medical LLMs. Furthermore, we introduce BiMed1.3M, an extensive Arabic-English bilingual instruction set covering 1.3 Million diverse medical interactions, resulting in over 632 million healthcare specialized tokens for instruction tuning. Our BiMed1.3M dataset includes 250k synthesized multi-turn doctor-patient chats and maintains a 1:2 Arabic-to-English ratio. Our model outperforms state-of-the-art Med42 and Meditron by average absolute gains of 2.5% and 4.1%, respectively, computed across multiple medical evaluation benchmarks in English, while operating at 8-times faster inference. Moreover, our BiMediX outperforms the generic Arabic-English bilingual LLM, Jais-30B, by average absolute gains of 10% on our Arabic medical benchmark and 15% on bilingual evaluations across multiple datasets. Our project page with source code and trained model is available at https://github.com/mbzuai-oryx/BiMediX .