mlsanityData to 14 Oct 2026

Reliability of Large Language Model Generated Clinical Reasoning in Assisted Reproductive Technology: Blinded Comparative Evaluation Study

arXiv 2510.16095 ↗First posted 17 Oct 2025cs.AIcited by 1

On the map: LLM reasoning → Clinical and biomedical applications

Abstract (arXiv)

Creating high-quality clinical Chains-of-Thought (CoTs) is crucial for explainable medical Artificial Intelligence (AI) while constrained by data scarcity. Although Large Language Models (LLMs) can synthesize medical data, their clinical reliability remains unverified. This study evaluates the reliability of LLM-generated CoTs and investigates prompting strategies to enhance their quality. In a blinded comparative study, senior clinicians in Assisted Reproductive Technology (ART) evaluated CoTs generated via three distinct strategies: Zero-shot, Random Few-shot (using shallow examples), and Selective Few-shot (using diverse, high-quality examples). These expert ratings were compared against evaluations from a state-of-the-art AI model (GPT-4o). The Selective Few-shot strategy significantly outperformed other strategies across all human evaluation metrics (p < .001). Critically, the Random Few-shot strategy offered no significant improvement over the Zero-shot baseline, demonstrating that low-quality examples are as ineffective as no examples. The success of the Selective strategy is attributed to two principles: "Gold-Standard Depth" (reasoning quality) and "Representative Diversity" (generalization). Notably, the AI evaluator failed to discern these critical performance differences. The clinical reliability of synthetic CoTs is dictated by strategic prompt curation, not the mere presence of examples. We propose a "Dual Principles" framework as a foundational methodology to generate trustworthy data at scale. This work offers a validated solution to the data bottleneck and confirms the indispensable role of human expertise in evaluating high-stakes clinical AI.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 24 of its 38 references that are in the map. First posted on arXiv 17 Oct 2025.

Builds on

24 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning8 Jun

Other references · 23

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  3. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  4. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  5. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 274
  6. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 263
  7. 2023Can Large Language Models Transform Computational Social Science?cited by 250
  8. 2023Data-centric Artificial Intelligence: A Surveycited by 203
  9. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 195
  10. 2019A deep learning system for differential diagnosis of skin diseasescited by 118
  11. 2023Towards Accurate Differential Diagnosis with Large Language Modelscited by 116
  12. 2023The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuningcited by 108
11 more
  1. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitationscited by 100
  2. 2024Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Modelscited by 99
  3. 2024Med42 -- Evaluating Fine-Tuning Strategies for Medical LLMs: Full-Parameter vs. Parameter-Efficient Approachescited by 52
  4. 2023Generating Faithful Synthetic Data with Large Language Models: A Case Study in Computational Social Sciencecited by 37
  5. 2023Taiyi: A Bilingual Fine-Tuned Large Language Model for Diverse Biomedical Taskscited by 36
  6. 2024Instance-adaptive Zero-shot Chain-of-Thought Promptingcited by 26
  7. 2023Multimodal Foundation Models Exploit Text to Make Medical Image Predictionscited by 20
  8. 2023Efficient Prompting via Dynamic In-Context Learningcited by 18
  9. 2024Evaluating the Application of ChatGPT in Outpatient Triage Guidance: A Comparative Studycited by 2
  10. 2024Enhancing Depression Diagnosis with Chain-of-Thought Promptingcited by 2
  11. 2025Evaluating the Feasibility and Accuracy of Large Language Models for Medical History-Taking in Obstetrics and Gynecologycited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025The Alignment Paradox of Medical Large Language Models in Infertility Care: Decoupling Algorithmic Improvement from Clinical Decision-making Qualitycited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Why Chain of Thought Fails in Clinical Text Understanding
  2. 2025Can Small and Reasoning Large Language Models Score Journal Articles for Research Quality and Do Averaging and Few-shot Help?
  3. 2025Evaluating Computational Accuracy of Large Language Models in Numerical Reasoning Tasks for Healthcare Applications
  4. 2024Enhancing Large Language Models for Clinical Decision Support by Incorporating Clinical Practice Guidelines
  5. 2024Zero-shot and Few-shot Generation Strategies for Artificial Clinical Records
  6. 2026To Reason or Not to: Selective Chain-of-Thought in Medical Question Answering
  7. 2023Methods to Estimate Large Language Model Confidence
  8. 2024Entry-level guide to the use of large language models for medical research