# Empowering Healthcare Practitioners with Language Models: Structuring Speech Transcripts in Two Real-World Clinical Applications

arXiv: https://arxiv.org/abs/2507.05517 · HTML: https://mlsanity.com/paper/2507.05517/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 41 references that are in the map.
- First posted on arXiv 7 Jul 2025.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [ACI-BENCH: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generation](https://mlsanity.com/paper/2306.02022/) (2023)
- [Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generation](https://mlsanity.com/paper/2403.06988/) (2024)
- [Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models](https://mlsanity.com/paper/2408.02442/) (2024)
- [Med42-v2: A Suite of Clinical LLMs](https://mlsanity.com/paper/2408.06142/) (2024)
- [Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again](https://mlsanity.com/paper/2203.08410/) (2022)
- [MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes](https://mlsanity.com/paper/2412.19260/) (2024)
- [NoteChat: A Dataset of Synthetic Doctor-Patient Conversations Conditioned on Clinical Notes](https://mlsanity.com/paper/2310.15959/) (2023)
- [Model Tuning or Prompt Tuning? A Study of Large Language Models for Clinical Concept and Relation Extraction](https://mlsanity.com/paper/2310.06239/) (2023)
- [From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyond](https://mlsanity.com/paper/2411.03590/) (2024)
- [Improving Open Information Extraction with Large Language Models: A Study on Demonstration Uncertainty](https://mlsanity.com/paper/2309.03433/) (2023)
- [IryoNLP at MEDIQA-CORR 2024: Tackling the Medical Error Detection & Correction Task On the Shoulders of Medical Agents](https://mlsanity.com/paper/2404.15488/) (2024)
- [Efficient extraction of medication information from clinical notes: an evaluation in two languages](https://mlsanity.com/paper/2502.03257/) (2025)

## Built on by (most cited first, 7 of 7)

- [MedMCP-Calc: Benchmarking LLMs for Realistic Medical Calculator Scenarios via MCP Integration](https://mlsanity.com/paper/2601.23049/) (2026)
- [Beyond Idealized Patients: Evaluating LLMs under Challenging Patient Behaviors in Medical Consultations](https://mlsanity.com/paper/2603.29373/) (2026)
- [Assessing Large Language Models for Structured Medical Order Extraction](https://mlsanity.com/paper/2510.10475/) (2025)
- [Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agents](https://mlsanity.com/paper/2510.17491/) (2025)
- [Overview of the MEDIQA-OE 2025 Shared Task on Medical Order Extraction from Doctor-Patient Consultations](https://mlsanity.com/paper/2510.26974/) (2025)
- [Retrieval-Augmented Generation Based Nurse Observation Extraction](https://mlsanity.com/paper/2603.26046/) (2026)
- [Retrieval-Augmented Large Language Models for Schema-Constrained Clinical Information Extraction](https://mlsanity.com/paper/2605.15467/) (2026)

## Abstract (arXiv)

Large language models (LLMs) such as GPT-4o and o1 have demonstrated strong performance on clinical natural language processing (NLP) tasks across multiple medical benchmarks. Nonetheless, two high-impact NLP tasks - structured tabular reporting from nurse dictations and medical order extraction from doctor-patient consultations - remain underexplored due to data scarcity and sensitivity, despite active industry efforts. Practical solutions to these real-world clinical tasks can significantly reduce the documentation burden on healthcare providers, allowing greater focus on patient care. In this paper, we investigate these two challenging tasks using private and open-source clinical datasets, evaluating the performance of both open- and closed-weight LLMs, and analyzing their respective strengths and limitations. Furthermore, we propose an agentic pipeline for generating realistic, non-sensitive nurse dictations, enabling structured extraction of clinical observations. To support further research in both areas, we release SYNUR and SIMORD, the first open-source datasets for nurse observation extraction and medical order extraction.