# Jointly Extracting Interventions, Outcomes, and Findings from RCT Reports with LLMs

arXiv: https://arxiv.org/abs/2305.03642 · HTML: https://mlsanity.com/paper/2305.03642/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 50 references that are in the map.
- First posted on arXiv 5 May 2023.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Entity, Relation, and Event Extraction with Contextualized Span Representations](https://mlsanity.com/paper/1909.03546/) (2019)
- [Structured Prediction as Translation between Augmented Natural Languages](https://mlsanity.com/paper/2101.05779/) (2021)
- [End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures](https://mlsanity.com/paper/1601.00770/) (2016)
- [Span-based Joint Entity and Relation Extraction with Transformer Pre-training](https://mlsanity.com/paper/1909.07755/) (2019)
- [Joint Extraction of Entities and Relations Based on a Novel Tagging Scheme](https://mlsanity.com/paper/1706.05075/) (2017)
- [Two are Better than One: Joint Entity and Relation Extraction with Table-Sequence Encoders](https://mlsanity.com/paper/2010.03851/) (2020)
- [A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature](https://mlsanity.com/paper/1806.04185/) (2018)
- [Effective Modeling of Encoder-Decoder Architecture for Joint Entity and Relation Extraction](https://mlsanity.com/paper/1911.09886/) (2019)
- [Inferring Which Medical Treatments Work from Reports of Clinical Trials](https://mlsanity.com/paper/1904.01606/) (2019)
- [Joint entity recognition and relation extraction as a multi-head selection problem](https://mlsanity.com/paper/1804.07847/) (2018)
- [Simultaneously Self-Attending to All Mentions for Full-Abstract Biological Relation Extraction](https://mlsanity.com/paper/1802.10569/) (2018)
- [Evidence Inference 2.0: More Data, Better Models](https://mlsanity.com/paper/2005.04177/) (2020)
- [An End-to-end Model for Entity-level Relation Extraction using Multi-instance Learning](https://mlsanity.com/paper/2102.05980/) (2021)
- [Adversarial training for multi-context joint entity and relation extraction](https://mlsanity.com/paper/1808.06876/) (2018)
- [Literature-Augmented Clinical Outcome Prediction](https://mlsanity.com/paper/2111.08374/) (2021)
- [Understanding Clinical Trial Reports: Extracting Medical Entities and Their Relations](https://mlsanity.com/paper/2010.03550/) (2020)
- [A Study on Agreement in PICO Span Annotations](https://mlsanity.com/paper/1904.09557/) (2019)
- [Data Mining in Clinical Trial Text: Transformers for Classification and Question Answering Tasks](https://mlsanity.com/paper/2001.11268/) (2020)

## Built on by (most cited first, 10 of 10)

- [Open (Clinical) LLMs are Sensitive to Instruction Phrasings](https://mlsanity.com/paper/2407.09429/) (2024)
- [PubMed and Beyond: Biomedical Literature Search in the Age of Artificial Intelligence](https://mlsanity.com/paper/2307.09683/) (2023)
- [Accelerating Clinical Evidence Synthesis with Large Language Models](https://mlsanity.com/paper/2406.17755/) (2024)
- [A foundation model for human-AI collaboration in medical literature mining](https://mlsanity.com/paper/2501.16255/) (2025)
- [Automatically Extracting Numerical Results from Randomized Controlled Trials with Large Language Models](https://mlsanity.com/paper/2405.01686/) (2024)
- [FactEHR: A Dataset for Evaluating Factuality in Clinical Notes Using LLMs](https://mlsanity.com/paper/2412.12422/) (2024)
- [Measuring Risk of Bias in Biomedical Reports: The RoBBR Benchmark](https://mlsanity.com/paper/2411.18831/) (2024)
- [Selective Fine-tuning on LLM-labeled Data May Reduce Reliance on Human Annotation: A Case Study Using Schedule-of-Event Table Detection](https://mlsanity.com/paper/2405.06093/) (2024)
- [EvidenceBench: A Benchmark for Extracting Evidence from Biomedical Papers](https://mlsanity.com/paper/2504.18736/) (2025)
- [Predicting Causal Effects from Natural Language Queries using Structured Representations](https://mlsanity.com/paper/2605.29631/) (2026)

## Abstract (arXiv)

Results from Randomized Controlled Trials (RCTs) establish the comparative effectiveness of interventions, and are in turn critical inputs for evidence-based care. However, results from RCTs are presented in (often unstructured) natural language articles describing the design, execution, and outcomes of trials; clinicians must manually extract findings pertaining to interventions and outcomes of interest from such articles. This onerous manual process has motivated work on (semi-)automating extraction of structured evidence from trial reports. In this work we propose and evaluate a text-to-text model built on instruction-tuned Large Language Models (LLMs) to jointly extract Interventions, Outcomes, and Comparators (ICO elements) from clinical abstracts, and infer the associated results reported. Manual (expert) and automated evaluations indicate that framing evidence extraction as a conditional generation task and fine-tuning LLMs for this purpose realizes considerable (~20 point absolute F1 score) gains over the previous SOTA. We perform ablations and error analyses to assess aspects that contribute to model performance, and to highlight potential directions for further improvements. We apply our model to a collection of published RCTs through mid-2022, and release a searchable database of structured findings: http://ico-relations.ebm-nlp.com