# ChiMed-GPT: A Chinese Medical Large Language Model with Full Training Regime and Better Alignment to Human Preferences

arXiv: https://arxiv.org/abs/2311.06025 · HTML: https://mlsanity.com/paper/2311.06025/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Placed in Medical LLM alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 30 of its 60 references that are in the map.
- First posted on arXiv 10 Nov 2023.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Medical LLM alignment](https://mlsanity.com/alignment/medical-llm-alignment/)

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [CMMLU: Measuring massive multitask language understanding in Chinese](https://mlsanity.com/paper/2306.09212/) (2023)
- [ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge](https://mlsanity.com/paper/2303.14070/) (2023)
- [MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data](https://mlsanity.com/paper/2304.08247/) (2023)
- [From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models](https://mlsanity.com/paper/2305.08283/) (2023)
- [Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models](https://mlsanity.com/paper/2305.18189/) (2023)
- [CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibility](https://mlsanity.com/paper/2307.09705/) (2023)
- [Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligence](https://mlsanity.com/paper/2209.02970/) (2022)
- [ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations](https://mlsanity.com/paper/1911.00720/) (2019)
- [ZEN 2.0: Continue Training and Adaption for N-gram Enhanced Text Encoders](https://mlsanity.com/paper/2105.01279/) (2021)
- [Ziya2: Data-centric Learning is All LLMs Need](https://mlsanity.com/paper/2311.03301/) (2023)
- [hyperdoc2vec: Distributed Representations of Hypertext Documents](https://mlsanity.com/paper/1805.03793/) (2018)
- [Overview of the CCKS 2019 Knowledge Graph Evaluation Track: Entity, Relation, Event and QA](https://mlsanity.com/paper/2003.03875/) (2020)

## Built on by (most cited first, 30 of 43)

- [From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discovery](https://mlsanity.com/paper/2508.14111/) (2025)
- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [Taiyi: A Bilingual Fine-Tuned Large Language Model for Diverse Biomedical Tasks](https://mlsanity.com/paper/2311.11608/) (2023)
- [A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directions](https://mlsanity.com/paper/2406.03712/) (2024)
- [FineMedLM-o1: Enhancing Medical Knowledge Reasoning Ability of LLM from Supervised Fine-Tuning to Test-Time Training](https://mlsanity.com/paper/2501.09213/) (2025)
- [Large Language Model Benchmarks in Medical Tasks](https://mlsanity.com/paper/2410.21348/) (2024)
- [M-IFEval: Multilingual Instruction-Following Evaluation](https://mlsanity.com/paper/2502.04688/) (2025)
- [A Survey on Large Language Models from General Purpose to Medical Applications: Datasets, Methodologies, and Evaluations](https://mlsanity.com/paper/2406.10303/) (2024)
- [3D-CT-GPT: Generating 3D Radiology Reports through Integration of Large Vision-Language Models](https://mlsanity.com/paper/2409.19330/) (2024)
- [MedHallBench: A New Benchmark for Assessing Hallucination in Medical Large Language Models](https://mlsanity.com/paper/2412.18947/) (2024)
- [Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoning](https://mlsanity.com/paper/2512.00818/) (2025)
- [BianCang: A Traditional Chinese Medicine Large Language Model](https://mlsanity.com/paper/2411.11027/) (2024)
- [Rethinking Diverse Human Preference Learning through Principal Component Analysis](https://mlsanity.com/paper/2502.13131/) (2025)
- [Multimodal Conditioned Diffusive Time Series Forecasting](https://mlsanity.com/paper/2504.19669/) (2025)
- [AutoMIR: Effective Zero-Shot Medical Information Retrieval without Relevance Labels](https://mlsanity.com/paper/2410.20050/) (2024)
- [Exploring the Inquiry-Diagnosis Relationship with Advanced Patient Simulators](https://mlsanity.com/paper/2501.09484/) (2025)
- [Aqulia-Med LLM: Pioneering Full-Process Open-Source Medical Language Models](https://mlsanity.com/paper/2406.12182/) (2024)
- [Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Models](https://mlsanity.com/paper/2411.06272/) (2024)
- [Odysseus Navigates the Sirens' Song: Dynamic Focus Decoding for Factual and Diverse Open-Ended Text Generation](https://mlsanity.com/paper/2503.08057/) (2025)
- [Respond to Change with Constancy: Instruction-tuning with LLM for Non-I.I.D. Network Traffic Classification](https://mlsanity.com/paper/2505.20866/) (2025)
- [Developing ChatGPT for Biology and Medicine: A Complete Review of Biomedical Question Answering](https://mlsanity.com/paper/2401.07510/) (2024)
- [Affective-ROPTester: Capability and Bias Analysis of LLMs in Predicting Retinopathy of Prematurity](https://mlsanity.com/paper/2507.05816/) (2025)
- [Training Data Selection with Gradient Orthogonality for Efficient Domain Adaptation](https://mlsanity.com/paper/2602.06359/) (2026)
- [DALL-M: Context-Aware Clinical Data Augmentation with LLMs](https://mlsanity.com/paper/2407.08227/) (2024)
- [CareBot: A Pioneering Full-Process Open-Source Medical Language Model](https://mlsanity.com/paper/2412.15236/) (2024)
- [DDO: Dual-Decision Optimization for LLM-Based Medical Consultation via Multi-Agent Collaboration](https://mlsanity.com/paper/2505.18630/) (2025)
- [KAG-Thinker: Interactive Thinking and Deep Reasoning in LLMs via Knowledge-Augmented Generation](https://mlsanity.com/paper/2506.17728/) (2025)
- [Recurrent Visual Feature Extraction and Stereo Attentions for CT Report Generation](https://mlsanity.com/paper/2506.19665/) (2025)
- [PclGPT: A Large Language Model for Patronizing and Condescending Language Detection](https://mlsanity.com/paper/2410.00361/) (2024)
- [Reviewing Clinical Knowledge in Medical Large Language Models: Training and Beyond](https://mlsanity.com/paper/2502.20988/) (2025)

## Abstract (arXiv)

Recently, the increasing demand for superior medical services has highlighted the discrepancies in the medical infrastructure. With big data, especially texts, forming the foundation of medical services, there is an exigent need for effective natural language processing (NLP) solutions tailored to the healthcare domain. Conventional approaches leveraging pre-trained models present promising results in this domain and current large language models (LLMs) offer advanced foundation for medical text processing. However, most medical LLMs are trained only with supervised fine-tuning (SFT), even though it efficiently empowers LLMs to understand and respond to medical instructions but is ineffective in learning domain knowledge and aligning with human preference. In this work, we propose ChiMed-GPT, a new benchmark LLM designed explicitly for Chinese medical domain, and undergoes a comprehensive training regime with pre-training, SFT, and RLHF. Evaluations on tasks including information extraction, question answering, and dialogue generation demonstrate ChiMed-GPT's superior performance over general domain LLMs. Furthermore, we analyze possible biases through prompting ChiMed-GPT to perform attitude scales regarding discrimination of patients, so as to contribute to further responsible development of LLMs in the medical domain. The code and model are released at https://github.com/synlp/ChiMed-GPT.