mlsanityData to 7 Oct 2026

RuleAlign: Making Large Language Models Better Physicians with Diagnostic Rule Alignment

arXiv 2408.12579 ↗First posted 22 Aug 2024cs.CL cs.AI cs.HCcited by 3

On the map: LLM alignment / RLHF / preference optimisation → Medical LLM alignment

Abstract (arXiv)

Large Language Models (LLMs) like GPT-4, MedPaLM-2, and Med-Gemini achieve performance competitively with human experts across various medical benchmarks. However, they still face challenges in making professional diagnoses akin to physicians, particularly in efficiently gathering patient information and reasoning the final diagnosis. To this end, we introduce the RuleAlign framework, designed to align LLMs with specific diagnostic rules. We develop a medical dialogue dataset comprising rule-based communications between patients and physicians and design an alignment learning approach through preference learning. Experimental results demonstrate the effectiveness of the proposed approach. We hope that our work can serve as an inspiration for exploring the potential of LLMs as AI physicians.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Medical LLM alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 22 of its 42 references that are in the map. First posted on arXiv 22 Aug 2024.

Builds on

22 of 42 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Integrating Physician Diagnostic Logic into Large Language Models: Preference Learning from Process Feedback11 Jan

Other references · 21

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  4. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  5. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  6. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  7. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  8. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  9. 2024Capabilities of Gemini Models in Medicinecited by 231
  10. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  11. 2023DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Taskcited by 130
  12. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
9 more
  1. 2023HuatuoGPT-II, One-stage Training for Medical Adaption of LLMscited by 91
  2. 2023DISC-MedLLM: Bridging General Large Language Models and Real-World Medical Consultationcited by 89
  3. 2024Towards Conversational Diagnostic AIcited by 79
  4. 2023BianQue: Balancing the Questioning and Suggestion Ability of Health LLMs with Multi-turn Health Conversations Polished by ChatGPTcited by 64
  5. 2024A Survey on Human Preference Learning for Large Language Modelscited by 23
  6. 2024SoFA: Shielded On-the-fly Alignment via Priority Rule Followingcited by 18
  7. 2024Towards Automatic Evaluation for LLMs' Clinical Capabilities: Metric, Data, and Algorithmcited by 14
  8. 2023RJUA-QA: A Comprehensive QA Dataset for Urologycited by 6
  9. 2024ASGEA: Exploiting Logic Rules from Align-Subgraphs for Entity Alignmentcited by 5

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2025Improve Rule Retrieval and Reasoning with Self-Induction and Relevance ReEstimatecited by 3
  2. 2024MentalArena: Self-play Training of Language Models for Diagnosis and Treatment of Mental Health Disorderscited by 1
  3. 2025Initial Investigation of LLM-Assisted Development of Rule-Based Clinical NLP Systemcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025IIMedGPT: Promoting Large Language Model Capabilities of Medical Tasks by Efficient Human Preference Alignment
  2. 2024MedCare: Advancing Medical LLMs through Decoupling Clinical Alignment and Knowledge Aggregation
  3. 2026ClinAlign: Scaling Healthcare Alignment from Clinician Preference
  4. 2023Aligning Large Language Models for Clinical Tasks
  5. 2026MedDialogRubrics: A Comprehensive Benchmark and Evaluation Framework for Multi-turn Medical Consultations in Large Language Models
  6. 2024Aligning (Medical) LLMs for (Counterfactual) Fairness
  7. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine
  8. 2025Enabling Doctor-Centric Medical AI with LLMs through Workflow-Aligned Tasks and Benchmarks