# LingYi: Medical Conversational Question Answering System based on Multi-modal Knowledge Graphs

arXiv: https://arxiv.org/abs/2204.09220 · HTML: https://mlsanity.com/paper/2204.09220/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 41 references that are in the map.
- First posted on arXiv 20 Apr 2022.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Neural Approaches to Conversational AI](https://mlsanity.com/paper/1809.08267/) (2018)
- [MedDialog: Two Large-scale Medical Dialogue Datasets](https://mlsanity.com/paper/2004.03329/) (2020)
- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation](https://mlsanity.com/paper/2109.05729/) (2021)
- [CRSLab: An Open-Source Toolkit for Building Conversational Recommender System](https://mlsanity.com/paper/2101.00939/) (2021)
- [COVID-19 Literature Knowledge Graph Construction and Drug Repurposing Report Generation](https://mlsanity.com/paper/2007.00576/) (2020)
- [Conversational Question Answering: A Survey](https://mlsanity.com/paper/2106.00874/) (2021)
- [Graph-Evolving Meta-Learning for Low-Resource Medical Dialogue Generation](https://mlsanity.com/paper/2012.11988/) (2020)
- [SMedBERT: A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text Mining](https://mlsanity.com/paper/2108.08983/) (2021)
- [Instance-aware Prompt Learning for Language Understanding and Generation](https://mlsanity.com/paper/2201.07126/) (2022)
- [On the Generation of Medical Dialogues for COVID-19](https://mlsanity.com/paper/2005.05442/) (2020)
- [DiSCoL: Toward Engaging Dialogue Systems through Conversational Line Guided Response Generation](https://mlsanity.com/paper/2102.02191/) (2021)
- [BERT-based Acronym Disambiguation with Multiple Training Strategies](https://mlsanity.com/paper/2103.00488/) (2021)
- [Leveraging Domain Agnostic and Specific Knowledge for Acronym Disambiguation](https://mlsanity.com/paper/2107.00316/) (2021)
- [SimCLAD: A Simple Framework for Contrastive Learning of Acronym Disambiguation](https://mlsanity.com/paper/2111.14306/) (2021)
- [ADBCMM : Acronym Disambiguation by Building Counterfactuals and Multilingual Mixing](https://mlsanity.com/paper/2112.08991/) (2021)

## Built on by (most cited first, 15 of 15)

- [Attribute-Consistent Knowledge Graph Representation Learning for Multi-Modal Entity Alignment](https://mlsanity.com/paper/2304.01563/) (2023)
- [T-RAG: Lessons from the LLM Trenches](https://mlsanity.com/paper/2402.07483/) (2024)
- [Mastering Symbolic Operations: Augmenting Language Models with Compiled Neural Networks](https://mlsanity.com/paper/2304.01665/) (2023)
- [Patchwork Learning: A Paradigm Towards Integrative Analysis across Diverse Biomedical Data Sources](https://mlsanity.com/paper/2305.06217/) (2023)
- [Does Knowledge Localization Hold True? Surprising Differences Between Entity and Relation Perspectives in Language Models](https://mlsanity.com/paper/2409.00617/) (2024)
- [Way to Specialist: Closing Loop Between Specialized LLM and Evolving Domain Knowledge Graph](https://mlsanity.com/paper/2411.19064/) (2024)
- [SeRTS: Self-Rewarding Tree Search for Biomedical Retrieval-Augmented Generation](https://mlsanity.com/paper/2406.11258/) (2024)
- [Overview of the NLPCC 2025 Shared Task 4: Multi-modal, Multilingual, and Multi-hop Medical Instructional Video Question Answering Challenge](https://mlsanity.com/paper/2505.06814/) (2025)
- [Review of Natural Language Processing in Pharmacology](https://mlsanity.com/paper/2208.10228/) (2022)
- [Graph AI in Medicine](https://mlsanity.com/paper/2310.13767/) (2023)
- [LMTuner: An user-friendly and highly-integrable Training Framework for fine-tuning Large Language Models](https://mlsanity.com/paper/2308.10252/) (2023)
- [Advancing Biomedicine with Graph Representation Learning: Recent Progress, Challenges, and Future Directions](https://mlsanity.com/paper/2306.10456/) (2023)
- [A Survey of the State-of-the-Art in Conversational Question Answering Systems](https://mlsanity.com/paper/2509.05716/) (2025)
- [MedGo: A Chinese Medical Large Language Model](https://mlsanity.com/paper/2410.20428/) (2024)
- [Towards Multimodal Graph Large Language Model](https://mlsanity.com/paper/2506.09738/) (2025)

## Abstract (arXiv)

The medical conversational system can relieve the burden of doctors and improve the efficiency of healthcare, especially during the pandemic. This paper presents a medical conversational question answering (CQA) system based on the multi-modal knowledge graph, namely "LingYi", which is designed as a pipeline framework to maintain high flexibility. Our system utilizes automated medical procedures including medical triage, consultation, image-text drug recommendation and record. To conduct knowledge-grounded dialogues with patients, we first construct a Chinese Medical Multi-Modal Knowledge Graph (CM3KG) and collect a large-scale Chinese Medical CQA (CMCQA) dataset. Compared with the other existing medical question-answering systems, our system adopts several state-of-the-art technologies including medical entity disambiguation and medical dialogue generation, which is more friendly to provide medical services to patients. In addition, we have open-sourced our codes which contain back-end models and front-end web pages at https://github.com/WENGSYX/LingYi. The datasets including CM3KG at https://github.com/WENGSYX/CM3KG and CMCQA at https://github.com/WENGSYX/CMCQA are also released to further promote future research.