mlsanityData to 14 Oct 2026

Exploring LLM-based Data Annotation Strategies for Medical Dialogue Preference Alignment

arXiv 2410.04112 ↗First posted 5 Oct 2024cs.CLcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Medical LLM alignment

Abstract (arXiv)

This research examines the use of Reinforcement Learning from AI Feedback (RLAIF) techniques to improve healthcare dialogue models, with the aim of tackling the challenges of preference-aligned data annotation while reducing the reliance on medical experts. We argue that the primary challenges in current RLAIF research for healthcare are the limitations of automated evaluation methods and the difficulties in accurately representing physician preferences. To address these challenges, we present a new evaluation framework based on standardized patient examinations. This framework is designed to objectively assess the effectiveness of large language models (LLMs) in guiding users and following instructions, enabling a comprehensive comparison across different models. Furthermore, our investigation of effective ways to express physician preferences using Constitutional AI algorithms highlighted the particular effectiveness of flowcharts. Utilizing this finding, we introduce an innovative agent-based approach for annotating preference data. This approach autonomously creates medical dialogue flows tailored to the patient's condition, demonstrates strong generalization abilities, and reduces the need for expert involvement. Our results show that the agent-based approach outperforms existing RLAIF annotation methods in standardized patient examinations and surpasses current open source medical dialogue LLMs in various test scenarios.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Medical LLM alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 41 references that are in the map. First posted on arXiv 5 Oct 2024.

Builds on

27 of 41 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Integrating Physician Diagnostic Logic into Large Language Models: Preference Learning from Process Feedback11 Jan
  2. 2023Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-world Multi-turn Dialogue7 Aug
  3. 2023HuatuoGPT, towards Taming Language Model to Be a Doctor24 May

Other references · 24

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2023Qwen Technical Reportcited by 3,344
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  5. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  6. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  7. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  8. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  9. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 286
  10. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 244
  11. 2023HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledgecited by 162
  12. 2020MedDialog: Two Large-scale Medical Dialogue Datasetscited by 161
12 more
  1. 2023Does Synthetic Data Generation of LLMs Help Clinical Text Mining?cited by 154
  2. 2023DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Taskcited by 131
  3. 2023CMB: A Comprehensive Medical Benchmark in Chinesecited by 115
  4. 2023HuatuoGPT-II, One-stage Training for Medical Adaption of LLMscited by 92
  5. 2023DISC-MedLLM: Bridging General Large Language Models and Real-World Medical Consultationcited by 90
  6. 2023ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluationcited by 88
  7. 2021Semantic Answer Similarity for Evaluating Question Answering Modelscited by 47
  8. 2020MedDG: An Entity-Centric Medical Consultation Dataset for Entity-Aware Medical Dialogue Generationcited by 45
  9. 2022A Benchmark for Automatic Medical Consultation System: Frameworks, Tasks and Datasetscited by 45
  10. 2024A Continued Pretrained LLM Approach for Automatic Medical Note Generationcited by 30
  11. 2024Effects of diversity incentives on sample diversity and downstream model performance in LLM-based text augmentationcited by 17
  12. 2023PlugMed: Improving Specificity in Patient-Centered Medical Dialogue Generation using In-Context Learningcited by 8

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025A Survey of LLM-based Agents in Medicine: How far are we from Baymax?cited by 85

Similar, not cited

Close in wording, no citation either way

  1. 2025IIMedGPT: Promoting Large Language Model Capabilities of Medical Tasks by Efficient Human Preference Alignment
  2. 2025Empathy by Design: Aligning Large Language Models for Healthcare Dialogue
  3. 2025Dialogue is Better Than Monologue: Instructing Medical LLMs via Strategical Conversations
  4. 2026MedDialogRubrics: A Comprehensive Benchmark and Evaluation Framework for Multi-turn Medical Consultations in Large Language Models
  5. 2025Reasoning LLMs in the Medical Domain: A Literature Survey
  6. 2025Aligning Large Language Models with Healthcare Stakeholders: A Pathway to Trustworthy AI Integration
  7. 2025AutoMedic: An Automated Evaluation Framework for Clinical Conversational Agents with Medical Dataset Grounding
  8. 2024Aqulia-Med LLM: Pioneering Full-Process Open-Source Medical Language Models