mlsanityData to 14 Oct 2026

ACI-BENCH: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generation

arXiv 2306.02022 ↗First posted 3 Jun 2023cs.CLcited by 118

Not on the map. Why?

Abstract (arXiv)

Recent immense breakthroughs in generative models such as in GPT4 have precipitated re-imagined ubiquitous usage of these models in all applications. One area that can benefit by improvements in artificial intelligence (AI) is healthcare. The note generation task from doctor-patient encounters, and its associated electronic medical record documentation, is one of the most arduous time-consuming tasks for physicians. It is also a natural prime potential beneficiary to advances in generative models. However with such advances, benchmarking is more critical than ever. Whether studying model weaknesses or developing new evaluation metrics, shared open datasets are an imperative part of understanding the current state-of-the-art. Unfortunately as clinic encounter conversations are not routinely recorded and are difficult to ethically share due to patient confidentiality, there are no sufficiently large clinic dialogue-note datasets to benchmark this task. Here we present the Ambient Clinical Intelligence Benchmark (ACI-BENCH) corpus, the largest dataset to date tackling the problem of AI-assisted note generation from visit dialogue. We also present the benchmark performances of several common state-of-the-art approaches.

Key facts. Cited by 118 papers in the map (data through 14 Oct 2026). Builds on 14 of its 48 references that are in the map. First posted on arXiv 3 Jun 2023.

Builds on

14 of 48 references.

References · 14

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2020Longformer: The Long-Document Transformercited by 3,413
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  6. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  7. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  8. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  9. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  10. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
  11. 2020Generating SOAP Notes from Doctor-Patient Conversations Using Modular Summarization Techniquescited by 64
  12. 2021Leveraging Pretrained Models for Automatic Summarization of Doctor-Patient Conversationscited by 42
2 more
  1. 2022PriMock57: A Dataset Of Primary Care Mock Consultationscited by 31
  2. 2020The Style-Content Duality of Attractiveness: Learning to Write Eye-Catching Headlines via Disentanglementcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 31; 2025: 36; 2026: 43.

118 papers.

Sort:

By area:

  1. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 246
  2. 2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 144
  3. 2024Med42-v2: A Suite of Clinical LLMscited by 90
  4. 2024AgentInstruct: Toward Generative Teaching with Agentic Flowscited by 89
  5. 2024RaTEScore: A Metric for Radiology Report Generationcited by 65
  6. 2023NoteChat: A Dataset of Synthetic Doctor-Patient Conversations Conditioned on Clinical Notescited by 48
  7. 2023Teaching Language Models to Hallucinate Less with Synthetic Taskscited by 31
  8. 2025BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLPcited by 28
  9. 2026HealthBench Professional: Evaluating Large Language Models on Real Clinician Chatscited by 26
  10. 2024MEDIC: Comprehensive Evaluation of Leading Indicators for LLM Safety and Utility in Clinical Applicationscited by 25
  11. 2025On Robustness and Reliability of Benchmark-Based Evaluation of LLMscited by 23
  12. 2024Comparing Two Model Designs for Clinical Note Generation; Is an LLM a Useful Evaluator of Consistency?cited by 20
38 more
  1. 2025Automatic Evaluation of Healthcare LLMs Beyond Question-Answeringcited by 20
  2. 2025A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignmentcited by 20
  3. 2024CriSPO: Multi-Aspect Critique-Suggestion-guided Automatic Prompt Optimization for Text Generationcited by 18
  4. 2023DocLens: Multi-aspect Fine-grained Evaluation for Medical Text Generationcited by 16
  5. 2023UMASS_BioNLP at MEDIQA-Chat 2023: Can LLMs generate high-quality synthetic note-oriented doctor-patient conversations?cited by 14
  6. 2023SummQA at MEDIQA-Chat 2023:In-Context Learning with GPT-4 for Medical Summarizationcited by 13
  7. 2025A Dataset for Addressing Patient's Information Needs related to Clinical Course of Hospitalizationcited by 13
  8. 2024Knowledge Planning in Large Language Models for Domain-Aligned Counseling Summarizationcited by 11
  9. 2024MedCodER: A Generative AI Assistant for Medical Codingcited by 11
  10. 2024Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluationcited by 11
  11. 2025Afrispeech-Dialog: A Benchmark Dataset for Spontaneous English Conversations in Healthcare and Beyondcited by 11
  12. 2024GenAudit: Fixing Factual Errors in Language Model Outputs with Evidencecited by 10
  13. 2024Towards Adapting Open-Source Large Language Models for Expert-Level Clinical Note Generationcited by 9
  14. 2024Knowledge Verification to Nip Hallucination in the Budcited by 8
  15. 2024A Perspective for Adapting Generalist AI to Specialized Medical AI Applications and Their Challengescited by 8
  16. 2026TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Explorationcited by 8
  17. 2023Improving Summarization with Human Editscited by 7
  18. 2024Exploring Robustness in Doctor-Patient Conversation Summarization: An Analysis of Out-of-Domain SOAP Notescited by 7
  19. 2024Improving Clinical Note Generation from Complex Doctor-Patient Conversationcited by 7
  20. 2025Empowering Healthcare Practitioners with Language Models: Structuring Speech Transcripts in Two Real-World Clinical Applicationscited by 7
  21. 2026Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMscited by 7
  22. 2023Extrinsically-Focused Evaluation of Omissions in Medical Summarizationcited by 6
  23. 2024WisPerMed at "Discharge Me!": Advancing Text Generation in Healthcare with Large Language Models, Dynamic Expert Selection, and Priming Techniques on MIMIC-IVcited by 6
  24. 2024Best Practices for Large Language Models in Radiologycited by 5
  25. 2024Process-Supervised Reward Models for Verifying Clinical Note Generation: A Scalable Approach Guided by Domain Expertisecited by 5
  26. 2025CLEAR: A Clinically-Grounded Tabular Framework for Radiology Report Evaluationcited by 5
  27. 2025Toward Reliable Clinical Coding with Language Models: Verification and Lightweight Adaptationcited by 5
  28. 2025TeamPath: Building MultiModal Pathology Experts with Reasoning AI Copilotscited by 5
  29. 2024Shimo Lab at "Discharge Me!": Discharge Summarization by Prompt-Driven Concatenation of Electronic Health Record Sectionscited by 4
  30. 2024e-Health CSIRO at "Discharge Me!" 2024: Generating Discharge Summary Sections with Fine-tuned Language Modelscited by 4
  31. 2024UF-HOBI at "Discharge Me!": A Hybrid Solution for Discharge Summary Generation Through Prompt-based Tuning of GatorTronGPT Modelscited by 4
  32. 2024Toward Relieving Clinician Burden by Automatically Generating Progress Notes using Interim Hospital Datacited by 4
  33. 2024BioMistral-NLU: Towards More Generalizable Medical Language Understanding through Instruction Tuningcited by 4
  34. 2025From Feedback to Checklists: Grounded Evaluation of AI-Generated Clinical Notescited by 4
  35. 2025Understanding the Effects of Domain Finetuning on LLMscited by 4
  36. 2026Can LLM Agents Generate Real-World Evidence? Evaluating Observational Studies in Medical Databasescited by 4
  37. 2026Generating Synthetic Doctor-Patient Conversations for Long-form Audio Summarizationcited by 4
  38. 2026MedLoCoMo: A Long-Context Multi-Session Medical Dialogue Benchmark for Large Language Modelscited by 4

50 most cited of 118 All 118 (JSON)