mlsanityData to 14 Oct 2026

Evaluation of Clinical Trials Reporting Quality using Large Language Models

arXiv 2510.04338 ↗First posted 5 Oct 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Reporting quality is an important topic in clinical trial research articles, as it can impact clinical decisions. In this article, we test the ability of large language models to assess the reporting quality of this type of article using the Consolidated Standards of Reporting Trials (CONSORT). We create CONSORT-QA, an evaluation corpus from two studies on abstract reporting quality with CONSORT-abstract standards. We then evaluate the ability of different large generative language models (from the general domain or adapted to the biomedical domain) to correctly assess CONSORT criteria with different known prompting methods, including Chain-of-thought. Our best combination of model and prompting method achieves 85% accuracy. Using Chain-of-thought adds valuable information on the model's reasoning for completing the task.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 12 of its 41 references that are in the map. First posted on arXiv 5 Oct 2025.

Builds on

12 of 41 references.

References · 12

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  7. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  8. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  9. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  10. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 538
  11. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 301
  12. 2019ScispaCy: Fast and Robust Models for Biomedical Natural Language Processingcited by 244

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating the Ability of Large Language Models to Identify Adherence to CONSORT Reporting Guidelines in Randomized Controlled Trials: A Methodological Evaluation Study
  2. 2026A primer on evaluation methods for large language models in healthcare
  3. 2025Evaluating Large Language Models for Evidence-Based Clinical Question Answering
  4. 2025Can Large Language Models Match the Conclusions of Systematic Reviews?
  5. 2025Reassessing Large Language Model Boolean Query Generation for Systematic Reviews
  6. 2025Exploring Large Language Models in Healthcare: Insights into Corpora Sources, Customization Strategies, and Evaluation Metrics
  7. 2025Developing Large Language Models for Clinical Research Using One Million Clinical Trials
  8. 2025Why Chain of Thought Fails in Clinical Text Understanding