# ACR: A Benchmark for Automatic Cohort Retrieval

arXiv: https://arxiv.org/abs/2406.14780 · HTML: https://mlsanity.com/paper/2406.14780/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 65 references that are in the map.
- First posted on arXiv 20 Jun 2024.

## Builds on (most cited first, 25 of 25)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [YaRN: Efficient Context Window Extension of Large Language Models](https://mlsanity.com/paper/2309.00071/) (2023)
- [Large Language Models for Information Retrieval: A Survey](https://mlsanity.com/paper/2308.07107/) (2023)
- [A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics](https://mlsanity.com/paper/2310.05694/) (2023)
- [Matching Patients to Clinical Trials with Large Language Models](https://mlsanity.com/paper/2307.15051/) (2023)
- [Scaling Clinical Trial Matching Using Large Language Models: A Case Study in Oncology](https://mlsanity.com/paper/2308.02180/) (2023)
- [Distilling Large Language Models for Matching Patients to Clinical Trials](https://mlsanity.com/paper/2312.09958/) (2023)
- [Understanding Clinical Trial Reports: Extracting Medical Entities and Their Relations](https://mlsanity.com/paper/2010.03550/) (2020)
- [A Scoping Review of Publicly Available Language Tasks in Clinical Natural Language Processing](https://mlsanity.com/paper/2112.05780/) (2021)
- [The Leaf Clinical Trials Corpus: a new resource for query generation from clinical trial eligibility criteria](https://mlsanity.com/paper/2207.13757/) (2022)
- [Effective Matching of Patients to Clinical Trials using Entity Extraction and Neural Re-ranking](https://mlsanity.com/paper/2307.00381/) (2023)
- [Patient Cohort Retrieval using Transformer Language Models](https://mlsanity.com/paper/2009.05121/) (2020)
- [Attention-Based LSTM Network for COVID-19 Clinical Trial Parsing](https://mlsanity.com/paper/2012.10063/) (2020)
- [CREATE: Cohort Retrieval Enhanced by Analysis of Text from Electronic Health Records using OMOP Common Data Model](https://mlsanity.com/paper/1901.07601/) (2019)
- [Model-assisted cohort selection with bias analysis for generating large-scale cohorts from the EHR for oncology research](https://mlsanity.com/paper/2001.09765/) (2020)
- [Towards Fair Patient-Trial Matching via Patient-Criterion Level Fairness Constraint](https://mlsanity.com/paper/2303.13790/) (2023)
- [Clinical Trial Recommendations Using Semantics-Based Inductive Inference and Knowledge Graph Embeddings](https://mlsanity.com/paper/2309.15979/) (2023)
- [Fair and Useful Cohort Selection](https://mlsanity.com/paper/2009.02207/) (2020)
- [Leximax Approximations and Representative Cohort Selection](https://mlsanity.com/paper/2205.01157/) (2022)
- [TREEMENT: Interpretable Patient-Trial Matching via Personalized Dynamic Tree-Based Memory Network](https://mlsanity.com/paper/2307.09942/) (2023)
- [Health Analytics: a systematic review of approaches to detect phenotype cohorts using electronic health records](https://mlsanity.com/paper/1707.07425/) (2017)
- [Coupling Symbolic Reasoning with Language Modeling for Efficient Longitudinal Understanding of Unstructured Electronic Medical Records](https://mlsanity.com/paper/2308.03360/) (2023)

## Built on by (most cited first, 2 of 2)

- [The Knowledge-Reasoning Dissociation: Fundamental Limitations of LLMs in Clinical Natural Language Inference](https://mlsanity.com/paper/2508.10777/) (2025)
- [Enhancing Hepatopathy Clinical Trial Efficiency: A Secure, Large Language Model-Powered Pre-Screening Pipeline](https://mlsanity.com/paper/2502.18531/) (2025)

## Abstract (arXiv)

Identifying patient cohorts is fundamental to numerous healthcare tasks, including clinical trial recruitment and retrospective studies. Current cohort retrieval methods in healthcare organizations rely on automated queries of structured data combined with manual curation, which are time-consuming, labor-intensive, and often yield low-quality results. Recent advancements in large language models (LLMs) and information retrieval (IR) offer promising avenues to revolutionize these systems. Major challenges include managing extensive eligibility criteria and handling the longitudinal nature of unstructured Electronic Medical Records (EMRs) while ensuring that the solution remains cost-effective for real-world application. This paper introduces a new task, Automatic Cohort Retrieval (ACR), and evaluates the performance of LLMs and commercial, domain-specific neuro-symbolic approaches. We provide a benchmark task, a query dataset, an EMR dataset, and an evaluation framework. Our findings underscore the necessity for efficient, high-quality ACR systems capable of longitudinal reasoning across extensive patient databases.