mlsanityData to 7 Oct 2026

OrthoDoc: Multimodal Large Language Model for Assisting Diagnosis in Computed Tomography

arXiv 2409.09052 ↗First posted 30 Aug 2024eess.IV cs.AI cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Multimodal large language models (MLLMs) have achieved significant success in the general field of image processing. Their emerging task generalization and freeform conversational capabilities can greatly facilitate medical diagnostic assistance, helping patients better understand their conditions and enhancing doctor-patient trust. Computed Tomography (CT) is a non-invasive imaging technique used to capture the internal mechanisms of a patient's condition and is widely utilized. However, in past research, the complex textural features of this imaging data have made accurate interpretation by algorithms challenging, impeding the performance of general LLMs in diagnostic assistance. To address this, we developed OrthoDoc, a MLLM designed for CT diagnostics. OrthoDoc is trained on 120,000 CT images and diagnostic reports and includes a Retrieval-Augmented Generation (RAG) module capable of effectively mitigating model hallucinations. This module is informed by extensive medical literature, textbooks, and explanatory data. Thus, OrthoDoc not only processes complex CT images but also stores, understands, and reasons over medical knowledge and language. In extensive experiments, OrthoDoc outperforms commercial models led by GPT-4, demonstrating superior diagnostic capabilities and accuracy. Specifically, OrthoDoc significantly surpasses existing models in the diagnosis of common orthopedic conditions such as fractures, arthritis, and tumors. Additionally, OrthoDoc exhibits robust generalization and stability when handling rare and complex cases.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 27 references that are in the map. First posted on arXiv 30 Aug 2024.

Builds on

23 of 27 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024LiteGPT: Large Vision-Language Model for Joint Chest X-ray Localization and Classification Task16 Jul
  2. 2024D-Rax: Domain-specific Radiologic assistant leveraging multi-modal data and eXpert model predictions2 Jul
  3. 2024MMedAgent: Learning to Use Medical Tools with Multi-modal Agent2 Jul
  4. 2024SERPENT-VLM : Self-Refining Radiology Report Generation Using Vision Language Models27 Apr
  5. 2024Efficiency in Focus: LayerNorm as a Catalyst for Fine-tuning Medical Visual Language Pre-trained Models25 Apr

Other references · 18

  1. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  2. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
  3. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 487
  4. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  5. 2024FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Researchcited by 165
  6. 2023Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Imagescited by 132
  7. 2023MedViT: A Robust Vision Transformer for Generalized Medical Image Classificationcited by 89
  8. 2024MemLLM: Finetuning LLMs to Use An Explicit Read-Write Memorycited by 35
  9. 2024A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directionscited by 34
  10. 2019Clinical XLNet: Modeling Sequential Clinical Notes and Predicting Prolonged Mechanical Ventilationcited by 30
  11. 2024A Continued Pretrained LLM Approach for Automatic Medical Note Generationcited by 30
  12. 2024MiniGPT-Med: Large Language Model as a General Interface for Radiology Diagnosiscited by 28
6 more
  1. 2024RAGAR, Your Falsehood Radar: RAG-Augmented Reasoning for Political Fact-Checking using Multimodal Large Language Modelscited by 27
  2. 2021MedGPT: Medical Concept Prediction from Clinical Narrativescited by 24
  3. 2024Investigating Mysteries of CoT-Augmented Distillationcited by 15
  4. 2024Beyond Accuracy: Investigating Error Types in GPT-4 Responses to USMLE Questionscited by 5
  5. 2024One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Modelscited by 4
  6. 2024Probing the Efficacy of Federated Parameter-Efficient Fine-Tuning of Vision Transformers for Medical Image Classificationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Multimodal Large Language Models for Medicine: A Comprehensive Surveycited by 16
  2. 2025Diagnosing Shoulder Disorders Using Multimodal Large Language Models and Consumer-Grade Camerascited by 1
  3. 2026Concept-Enhanced Multimodal RAG: Towards Interpretable and Accurate Radiology Report Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Evaluating the Diagnostic Classification Ability of Multimodal Large Language Models: Insights from the Osteoarthritis Initiative
  2. 2023OphGLM: Training an Ophthalmology Large Language-and-Vision Assistant based on Instructions and Dialogue
  3. 2024Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology
  4. 2025MedErr-CT: A Visual Question Answering Benchmark for Identifying and Correcting Errors in CT Reports
  5. 2025MedBLINK: Probing Basic Perception in Multimodal Language Models for Medicine
  6. 2024On Large Visual Language Models for Medical Imaging Analysis: An Empirical Study
  7. 2025Evaluating Vision Language Models (VLMs) for Radiology: A Comprehensive Analysis
  8. 2026NV-Reason-CT: 3D Visual Language Model for CT Analysis