# MMIR-TCM: Memory-Integrated Multimodal Inference and Retrieval for TCM Clinical Decision Support

arXiv: https://arxiv.org/abs/2607.01814 · HTML: https://mlsanity.com/paper/2607.01814/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 52 references that are in the map.
- First posted on arXiv 2 Jul 2026.

## Builds on (most cited first, 25 of 25)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [Ragas: Automated Evaluation of Retrieval Augmented Generation](https://mlsanity.com/paper/2309.15217/) (2023)
- [The Faiss library](https://mlsanity.com/paper/2401.08281/) (2024)
- [A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Review](https://mlsanity.com/paper/2405.02559/) (2024)
- [Trustworthy clinical AI solutions: a unified review of uncertainty quantification in deep learning models for medical image analysis](https://mlsanity.com/paper/2210.03736/) (2022)
- [Constrained Decoding for Secure Code Generation](https://mlsanity.com/paper/2405.00218/) (2024)
- [Out-of-distribution Detection in Medical Image Analysis: A survey](https://mlsanity.com/paper/2404.18279/) (2024)
- [TCMBench: A Comprehensive Benchmark for Evaluating Large Language Models in Traditional Chinese Medicine](https://mlsanity.com/paper/2406.01126/) (2024)
- [BianCang: A Traditional Chinese Medicine Large Language Model](https://mlsanity.com/paper/2411.11027/) (2024)
- [Empirical Validation of Conformal Prediction for Trustworthy Skin Lesions Classification](https://mlsanity.com/paper/2312.07460/) (2023)
- [Enhancing Large Language Models for Clinical Decision Support by Incorporating Clinical Practice Guidelines](https://mlsanity.com/paper/2401.11120/) (2024)
- [MedCite: Can Language Models Generate Verifiable Text for Medicine?](https://mlsanity.com/paper/2506.06605/) (2025)
- [OpenTCM: A GraphRAG-Empowered LLM-based System for Traditional Chinese Medicine Knowledge Retrieval and Diagnosis](https://mlsanity.com/paper/2504.20118/) (2025)
- [TongueSAM: An Universal Tongue Segmentation Model Based on SAM with Zero-Shot](https://mlsanity.com/paper/2308.06444/) (2023)
- [TCMD: A Traditional Chinese Medicine QA Dataset for Evaluating Large Language Models](https://mlsanity.com/paper/2406.04941/) (2024)
- [GroUSE: A Benchmark to Evaluate Evaluators in Grounded Question Answering](https://mlsanity.com/paper/2409.06595/) (2024)
- [TCM-Ladder: A Benchmark for Multimodal Question Answering on Traditional Chinese Medicine](https://mlsanity.com/paper/2505.24063/) (2025)
- [TCM-FTP: Fine-Tuning Large Language Models for Herbal Prescription Prediction](https://mlsanity.com/paper/2407.10510/) (2024)
- [MTCMB: A Multi-Task Benchmark Framework for Evaluating LLMs on Knowledge, Reasoning, and Safety in Traditional Chinese Medicine](https://mlsanity.com/paper/2506.01252/) (2025)
- [Memory-SAM: Human-Prompt-Free Tongue Segmentation via Retrieval-to-Prompt](https://mlsanity.com/paper/2510.15849/) (2025)

## Abstract (arXiv)

Traditional Chinese Medicine (TCM) diagnosis, particularly through tongue inspection, faces persistent challenges in subjectivity and reproducibility. The application of multimodal artificial intelligence to TCM clinical tasks, such as syndrome differentiation and prescription generation, is significantly hampered by the semantic gap between visual tongue features and textual reasoning, as well as the lack of large-scale, standardized datasets. To address these challenges, we introduce MMIR-TCM, a novel framework that emulates the diagnostic process of TCM experts by integrating multimodal large language model(MLLM) with memory-augmented segmentation and retrieval-augmented generation (RAG). Employing a three-stage architecture, MMIR-TCM integrates a training-free Memory-SAM module for robust tongue extraction, a fine-tuned Qwen3-VL model for structured tongue diagnosis generation, and a Qwen3-based RAG component for evidence-grounded clinical decision support generation. The framework was developed and validated using MedTCM, a new large-scale multimodal dataset that we introduce specifically for advanced TCM research. To properly evaluate our framework's clinical accuracy, which existing metrics fail to capture, we also developed TDEU, a domain-specific evaluation metric incorporating semantic understanding and diagnostic importance. Our comprehensive experiments demonstrate that MMIR-TCM significantly outperforms leading models, including GPT-4o and Gemini 2.5 Flash.