# Small Language Models Learn Enhanced Reasoning Skills from Medical Textbooks

arXiv: https://arxiv.org/abs/2404.00376 · HTML: https://mlsanity.com/paper/2404.00376/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 24 of its 53 references that are in the map.
- First posted on arXiv 30 Mar 2024.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)

## Builds on (most cited first, 24 of 24)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [Towards Expert-Level Medical Question Answering with Large Language Models](https://mlsanity.com/paper/2305.09617/) (2023)
- [Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine](https://mlsanity.com/paper/2311.16452/) (2023)
- [How Language Model Hallucinations Can Snowball](https://mlsanity.com/paper/2305.13534/) (2023)
- [BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains](https://mlsanity.com/paper/2402.10373/) (2024)
- [UL2: Unifying Language Learning Paradigms](https://mlsanity.com/paper/2205.05131/) (2022)
- [MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data](https://mlsanity.com/paper/2304.08247/) (2023)
- [MEDITRON-70B: Scaling Medical Pretraining for Large Language Models](https://mlsanity.com/paper/2311.16079/) (2023)
- [Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions](https://mlsanity.com/paper/2402.18060/) (2024)
- [AlpaCare:Instruction-tuned Large Language Models for Medical Application](https://mlsanity.com/paper/2310.14558/) (2023)
- [PMC-LLaMA: Towards Building Open-source Language Models for Medicine](https://mlsanity.com/paper/2304.14454/) (2023)
- [Look at the First Sentence: Position Bias in Question Answering](https://mlsanity.com/paper/2004.14602/) (2020)
- [Towards Conversational Diagnostic AI](https://mlsanity.com/paper/2401.05654/) (2024)
- [Clinical Camel: An Open Expert-Level Medical Language Model with Dialogue-Based Knowledge Encoding](https://mlsanity.com/paper/2305.12031/) (2023)
- [K-QA: A Real-World Medical Q&A Benchmark](https://mlsanity.com/paper/2401.14493/) (2024)
- [Me LLaMA: Foundation Large Language Models for Medical Applications](https://mlsanity.com/paper/2402.12749/) (2024)

## Built on by (most cited first, 30 of 30)

- [UltraMedical: Building Specialized Generalists in Biomedicine](https://mlsanity.com/paper/2406.03949/) (2024)
- [AGILE: A Novel Reinforcement Learning Framework of LLM Agents](https://mlsanity.com/paper/2405.14751/) (2024)
- [Rationale-Guided Retrieval Augmented Generation for Medical Question Answering](https://mlsanity.com/paper/2411.00300/) (2024)
- [A Survey on Large Language Models from General Purpose to Medical Applications: Datasets, Methodologies, and Evaluations](https://mlsanity.com/paper/2406.10303/) (2024)
- [Multimodal Large Language Models for Medicine: A Comprehensive Survey](https://mlsanity.com/paper/2504.21051/) (2025)
- [Pattern Recognition or Medical Knowledge? The Problem with Multiple-Choice Questions in Medicine](https://mlsanity.com/paper/2406.02394/) (2024)
- [Bottom-up Domain-specific Superintelligence: A Reliable Knowledge Graph is What We Need](https://mlsanity.com/paper/2507.13966/) (2025)
- [Agentic Medical Knowledge Graphs Enhance Medical Question Answering: Bridging the Gap Between LLMs and Evolving Medical Knowledge](https://mlsanity.com/paper/2502.13010/) (2025)
- [SLM-SQL: An Exploration of Small Language Models for Text-to-SQL](https://mlsanity.com/paper/2507.22478/) (2025)
- [Large Model Strategic Thinking, Small Model Efficiency: Transferring Theory of Mind in Large Language Models](https://mlsanity.com/paper/2408.05241/) (2024)
- [MedConceptsQA: Open Source Medical Concepts QA Benchmark](https://mlsanity.com/paper/2405.07348/) (2024)
- [CUPCase: Clinically Uncommon Patient Cases and Diagnoses Dataset](https://mlsanity.com/paper/2503.06204/) (2025)
- [How Well Can General Vision-Language Models Learn Medicine By Watching Public Educational Videos?](https://mlsanity.com/paper/2504.14391/) (2025)
- [MACD: Multi-Agent Clinical Diagnosis with Self-Learned Knowledge for LLM](https://mlsanity.com/paper/2509.20067/) (2025)
- [Leveraging Online Data to Enhance Medical Knowledge in a Small Persian Language Model](https://mlsanity.com/paper/2505.16000/) (2025)
- [Routing with Generated Data: Annotation-Free LLM Skill Estimation and Expert Selection](https://mlsanity.com/paper/2601.09692/) (2026)
- [Semi-Supervised Preference Optimization with Limited Feedback](https://mlsanity.com/paper/2511.00040/) (2025)
- [Unveiling and Bridging the Functional Perception Gap in MLLMs: Atomic Visual Alignment and Hierarchical Evaluation via PET-Bench](https://mlsanity.com/paper/2601.02737/) (2026)
- [LAPIS: Language Model-Augmented Police Investigation System](https://mlsanity.com/paper/2407.20248/) (2024)
- [Small Language Models (SLMs) Can Still Pack a Punch: A survey (updated 2026)](https://mlsanity.com/paper/2501.05465/) (2025)
- [RAR2: Retrieval-Augmented Medical Reasoning via Thought-Driven Retrieval](https://mlsanity.com/paper/2509.22713/) (2025)
- [Lightweight Large Language Model for Medication Enquiry: Med-Pal](https://mlsanity.com/paper/2407.12822/) (2024)
- [TeamMedAgents: Pareto-Efficient Multi-Agent Medical Reasoning Through Teamwork Theory](https://mlsanity.com/paper/2508.08115/) (2025)
- [The Reliability Gap in Benchmark Auditing: Distribution Shift and Scale as Failure Modes of Contamination Detection](https://mlsanity.com/paper/2606.03305/) (2026)
- [A Locally Executable AI System for Improving Preoperative Patient Communication: A Multi-Domain Clinical Evaluation](https://mlsanity.com/paper/2510.01671/) (2025)
- [SOLVE-Med: Specialized Orchestration for Leading Vertical Experts across Medical Specialties](https://mlsanity.com/paper/2511.03542/) (2025)
- [Supervised Fine-tuning with Synthetic Rationale Data Hurts Real-World Disease Prediction](https://mlsanity.com/paper/2606.10279/) (2026)
- [MedJudgeRAG: Option-Wise Evidence Judgment with Dynamic Knowledge Graphs for Medical MCQA](https://mlsanity.com/paper/2607.24838/) (2026)
- [Gaokerena: A Small Persian Medical Language Model Family](https://mlsanity.com/paper/2608.00932/) (2026)
- [Patients-like-me: A Variational LM--GNN Framework for Explainable Clinical Prediction](https://mlsanity.com/paper/2608.04193/) (2026)

## Abstract (arXiv)

While recent advancements in commercial large language models (LM) have shown promising results in medical tasks, their closed-source nature poses significant privacy and security concerns, hindering their widespread use in the medical field. Despite efforts to create open-source models, their limited parameters often result in insufficient multi-step reasoning capabilities required for solving complex medical problems. To address this, we introduce Meerkat, a new family of medical AI systems ranging from 7 to 70 billion parameters. The models were trained using our new synthetic dataset consisting of high-quality chain-of-thought reasoning paths sourced from 18 medical textbooks, along with diverse instruction-following datasets. Our systems achieved remarkable accuracy across six medical benchmarks, surpassing the previous best models such as MediTron and BioMistral, and GPT-3.5 by a large margin. Notably, Meerkat-7B surpassed the passing threshold of the United States Medical Licensing Examination (USMLE) for the first time for a 7B-parameter model, while Meerkat-70B outperformed GPT-4 by an average of 1.3%. Additionally, Meerkat-70B correctly diagnosed 21 out of 38 complex clinical cases, outperforming humans' 13.8 and closely matching GPT-4's 21.8. Our systems offered more detailed free-form responses to clinical queries compared to existing small models, approaching the performance level of large commercial models. This significantly narrows the performance gap with large LMs, showcasing its effectiveness in addressing complex medical challenges.