# CollectiveSFT: Scaling Large Language Models for Chinese Medical Benchmark with Collective Instructions in Healthcare

arXiv: https://arxiv.org/abs/2407.19705 · HTML: https://mlsanity.com/paper/2407.19705/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 30 references that are in the map.
- First posted on arXiv 29 Jul 2024.

## Builds on (most cited first, 17 of 17)

- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering](https://mlsanity.com/paper/2203.14371/) (2022)
- [Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor](https://mlsanity.com/paper/2212.09689/) (2022)
- [HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge](https://mlsanity.com/paper/2304.06975/) (2023)
- [CMB: A Comprehensive Medical Benchmark in Chinese](https://mlsanity.com/paper/2308.08833/) (2023)
- [HEAD-QA: A Healthcare Dataset for Complex Reasoning](https://mlsanity.com/paper/1906.04701/) (2019)
- [HuatuoGPT-II, One-stage Training for Medical Adaption of LLMs](https://mlsanity.com/paper/2311.09774/) (2023)
- [DISC-MedLLM: Bridging General Large Language Models and Real-World Medical Consultation](https://mlsanity.com/paper/2308.14346/) (2023)
- [Building Emotional Support Chatbots in the Era of LLMs](https://mlsanity.com/paper/2308.11584/) (2023)
- [FrenchMedMCQA: A French Multiple-Choice Question Answering Dataset for Medical domain](https://mlsanity.com/paper/2304.04280/) (2023)
- [From Beginner to Expert: Modeling Medical Knowledge into General LLMs](https://mlsanity.com/paper/2312.01040/) (2023)

## Built on by (most cited first, 1 of 1)

- [Finetune Once: Decoupling General & Domain Learning with Dynamic Boosted Annealing](https://mlsanity.com/paper/2509.26242/) (2025)

## Abstract (arXiv)

The rapid progress in Large Language Models (LLMs) has prompted the creation of numerous benchmarks to evaluate their capabilities.This study focuses on the Comprehensive Medical Benchmark in Chinese (CMB), showcasing how dataset diversity and distribution in supervised fine-tuning (SFT) may enhance LLM performance.Remarkably, We successfully trained a smaller base model to achieve scores comparable to larger models, indicating that a diverse and well-distributed dataset can optimize performance regardless of model size.This study suggests that even smaller models may reach high performance levels with carefully curated and varied datasets. By integrating a wide range of instructional content, our approach addresses potential issues such as data quality inconsistencies. Our results imply that a broader spectrum of training data may enhance a model's ability to generalize and perform effectively across different medical scenarios, highlighting the importance of dataset quality and diversity in fine-tuning processes. We open-source the model for future research at https://github.com/CAS-SIAT-XinHai/CollectiveSFT