mlsanityData to 7 Oct 2026

HuatuoGPT-II, One-stage Training for Medical Adaption of LLMs

arXiv 2311.09774 ↗First posted 16 Nov 2023cs.CL cs.AI cs.LGcited by 91

Not on the map. Why?

Abstract (arXiv)

Adapting a language model into a specific domain, a.k.a `domain adaption', is a common practice when specialized knowledge, e.g. medicine, is not encapsulated in a general language model like Llama2. The challenge lies in the heterogeneity of data across the two training stages, as it varies in languages, genres, or formats. To tackle this and simplify the learning protocol, we propose to transform heterogeneous data, from the both pre-training and supervised stages, into a unified, simple input-output pair format. We validate the new protocol in the domains where proprietary LLMs like ChatGPT perform relatively poorly, such as Traditional Chinese Medicine. The developed model, HuatuoGPT-II, has shown state-of-the-art performance in Chinese medicine domain on a number of benchmarks, e.g. medical licensing exams. It even outperforms proprietary models like ChatGPT and GPT-4 in some aspects, especially in Traditional Chinese Medicine. Expert manual evaluations further validate HuatuoGPT-II's advantages over existing LLMs. Notably, HuatuoGPT-II was benchmarked in a fresh Chinese National Medical Licensing Examination where it achieved the best performance, showcasing not only its effectiveness but also its generalization capabilities.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Builds on 36 of its 49 references that are in the map. First posted on arXiv 16 Nov 2023.

Builds on

36 of 49 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2023Qwen Technical Reportcited by 3,273
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  8. 2023LIMA: Less Is More for Alignmentcited by 1,080
  9. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  10. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  11. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  12. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
24 more
  1. 2023Textbooks Are All You Needcited by 561
  2. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  3. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  4. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  5. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  6. 2023HuatuoGPT, towards Taming Language Model to Be a Doctorcited by 211
  7. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 199
  8. 2023HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledgecited by 161
  9. 2020MedDialog: Two Large-scale Medical Dialogue Datasetscited by 157
  10. 2023Self-Alignment with Instruction Backtranslationcited by 149
  11. 2023DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Taskcited by 130
  12. 2023Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-world Multi-turn Dialoguecited by 118
  13. 2023CMB: A Comprehensive Medical Benchmark in Chinesecited by 113
  14. 2021CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmarkcited by 99
  15. 2023Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Datasetcited by 90
  16. 2023DISC-MedLLM: Bridging General Large Language Models and Real-World Medical Consultationcited by 89
  17. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  18. 2023InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuningcited by 80
  19. 2023BianQue: Balancing the Questioning and Suggestion Ability of Health LLMs with Multi-turn Health Conversations Polished by ChatGPTcited by 64
  20. 2023ChatHaruhi: Reviving Anime Character in Reality via Large Language Modelcited by 57
  21. 2023Huatuo-26M, a Large-scale Chinese Medical QA Datasetcited by 46
  22. 2022Datasheet for the Pilecited by 41
  23. 2023Phoenix: Democratizing ChatGPT across Languagescited by 32
  24. 2022Consistency is the key to further mitigating catastrophic forgetting in continual learningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 41; 2025: 35; 2026: 15.

91 papers.

Sort:

By area:

  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  2. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 257
  3. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  4. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoningcited by 217
  5. 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scalecited by 194
  6. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  7. 2024AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulatorcited by 99
  8. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  9. 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 74
  10. 2024CoD, Towards an Interpretable Medical Agent using Chain of Diagnosiscited by 43
  11. 2024Apollo: A Lightweight Multilingual Medical LLM towards Democratizing Medical AI to 6B Peoplecited by 38
  12. 2024A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directionscited by 34
38 more
  1. 2025Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RLcited by 34
  2. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  3. 2024Pushing The Limit of LLM Capacity for Text Classificationcited by 28
  4. 2024ClinicalLab: Aligning Agents for Multi-Departmental Clinical Diagnostics in the Real Worldcited by 28
  5. 2024PsycoLLM: Enhancing LLM for Psychological Understanding and Evaluationcited by 26
  6. 2024RAG-Instruct: Boosting LLMs with Diverse Retrieval-Augmented Instructionscited by 25
  7. 2025Narrow Finetuning Leaves Clearly Readable Traces in Activation Differencescited by 24
  8. 2025FineMedLM-o1: Enhancing Medical Knowledge Reasoning Ability of LLM from Supervised Fine-Tuning to Test-Time Trainingcited by 19
  9. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  10. 2024Reasoning Like a Doctor: Improving Medical Dialogue Systems via Diagnostic Reasoning Process Alignmentcited by 17
  11. 2024Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Expertscited by 17
  12. 2024PediatricsGPT: Large Language Models as Chinese Medical Assistants for Pediatric Applicationscited by 16
  13. 2024A Survey on Large Language Models from General Purpose to Medical Applications: Datasets, Methodologies, and Evaluationscited by 16
  14. 2024MedAide: Information Fusion and Anatomy of Medical Intents via LLM-based Agent Collaborationcited by 16
  15. 2025Multimodal Large Language Models for Medicine: A Comprehensive Surveycited by 16
  16. 2024Towards Automatic Evaluation for LLMs' Clinical Capabilities: Metric, Data, and Algorithmcited by 14
  17. 2025Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoningcited by 14
  18. 2025KERAP: A Knowledge-Enhanced Reasoning Approach for Accurate Zero-shot Diagnosis Prediction Using Multi-agent LLMscited by 12
  19. 2024BianCang: A Traditional Chinese Medicine Large Language Modelcited by 11
  20. 2024Integrating Physician Diagnostic Logic into Large Language Models: Preference Learning from Process Feedbackcited by 9
  21. 2025MAP: Evaluation and Multi-Agent Enhancement of Large Language Models for Inpatient Pathwayscited by 9
  22. 2024TAIA: Large Language Models are Out-of-Distribution Data Learnerscited by 8
  23. 2024PIORS: Personalized Intelligent Outpatient Reception based on Large Language Model with Multi-Agents Medical Scenario Simulationcited by 8
  24. 2025Judge as A Judge: Improving the Evaluation of Retrieval-Augmented Generation through the Judge-Consistency of Large Language Modelscited by 8
  25. 2025MedReflect: Teaching Medical LLMs to Self-Improve via Reflective Correctioncited by 8
  26. 2024MING-MOE: Enhancing Medical Multi-Task Learning in Large Language Models with Sparse Mixture of Low-Rank Adapter Expertscited by 7
  27. 2024Beyond Fine-tuning: Unleashing the Potential of Continuous Pretraining for Clinical LLMscited by 7
  28. 2024LLMs Could Autonomously Learn Without External Supervisioncited by 6
  29. 2025Exploring Efficiency Frontiers of Thinking Budget in Medical Reasoning: Scaling Laws between Computational Resources and Reasoning Qualitycited by 6
  30. 2025ShizhenGPT: Towards Multimodal LLMs for Traditional Chinese Medicinecited by 6
  31. 2024Medical Dialogue: A Survey of Categories, Methods, Evaluation and Challengescited by 5
  32. 2024LLMs for Doctors: Leveraging Medical LLMs to Assist Doctors, Not Replace Themcited by 5
  33. 2025TCM-Ladder: A Benchmark for Multimodal Question Answering on Traditional Chinese Medicinecited by 5
  34. 2025Training-free LLM Merging for Multi-task Learningcited by 5
  35. 2025Inflated Excellence or True Performance? Rethinking Medical Diagnostic Benchmarks with Dynamic Evaluationcited by 5
  36. 2024MedCare: Advancing Medical LLMs through Decoupling Clinical Alignment and Knowledge Aggregationcited by 4
  37. 2024Fine-Tuning Medical Language Models for Enhanced Long-Contextual Understanding and Domain Expertisecited by 4
  38. 2025Structural Entropy Guided Agent for Detecting and Repairing Knowledge Deficiencies in LLMscited by 4

50 most cited of 91 All 91 (JSON)