mlsanityData to 14 Oct 2026

Guideline-as-Oracle: Zero-Annotation Training of an Ophthalmic Telephone Triage Agent

arXiv 2608.04772 ↗First posted 5 Aug 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Scaling supervision for multi-turn medical agents is difficult because expert dialogue annotation is costly and clinical conversations are privacy-restricted. We introduce Guideline-as-Oracle (GAO), which compiles American Academy of Ophthalmology guidance into a 70-row operational rule table and uses it as the sole source of instance-level supervision for 3,000 training dialogues, reserving human labeling for evaluation. Because converting rules into dialogues is itself a design problem, we catalog eight construction strategies, including cited-row tier assignment, one-fact boundary pairs, metadata-only repair, and label repair, and characterize the evidential status of each: labeling mechanism, null, confounded, or evaluated only as a package. Fine-tuning a 9B backbone on this corpus yields GAO-Triage, improving agreement with a 201-case operational reference from 61.7% to 74.1% (exact McNemar p=0.0046) and emergent-case recall from 9.5% to 69.0%; the gains persist across a second seed and patient simulator. None of the seven general-purpose systems we test dominates GAO-Triage on both metrics, and GAO-Triage requires no frontier model at inference time. Permuting label-dialogue assignments collapses the model to a constant-routine predictor, indicating that the signal lies in guideline-derived assignment rather than dialogue surface form. Label repair coincides with the disappearance of a late-training safety degradation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 28 references that are in the map. First posted on arXiv 5 Aug 2026.

Builds on

17 of 28 references.

References · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  7. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  8. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
  9. 2017Snorkel: Rapid Training Data Creation with Weak Supervisioncited by 372
  10. 2016Data Programming: Creating Large Training Sets, Quicklycited by 246
  11. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 231
  12. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
5 more
  1. 2018Neural User Simulation for Corpus-based Policy Optimisation for Spoken Dialogue Systemscited by 42
  2. 2025ALFA: Aligning LLMs to Ask Good Questions A Case Study in Clinical Reasoningcited by 25
  3. 2025MedResearcher-R1: Expert-Level Medical Deep Researcher via A Knowledge-Informed Trajectory Synthesis Frameworkcited by 22
  4. 2025Teaching Language Models To Gather Information Proactivelycited by 14
  5. 2025From Promising Capability to Pervasive Bias: Assessing Large Language Models for Emergency Department Triagecited by 5

Built on by

Not cited yet.