mlsanityData to 7 Oct 2026

CareMedEval dataset: Evaluating Critical Appraisal and Reasoning in the Biomedical Field

arXiv 2511.03441 ↗First posted 5 Nov 2025cs.CL cs.AIcited by 0

On the map: LLM reasoning → Clinical and biomedical applications

Abstract (arXiv)

Critical appraisal of scientific literature is an essential skill in the biomedical field. While large language models (LLMs) can offer promising support in this task, their reliability remains limited, particularly for critical reasoning in specialized domains. We introduce CareMedEval, an original dataset designed to evaluate LLMs on biomedical critical appraisal and reasoning tasks. Derived from authentic exams taken by French medical students, the dataset contains 534 questions based on 37 scientific articles. Unlike existing benchmarks, CareMedEval explicitly evaluates critical reading and reasoning grounded in scientific papers. Benchmarking state-of-the-art generalist and biomedical-specialized LLMs under various context conditions reveals the difficulty of the task: open and commercial models fail to exceed an Exact Match Rate of 0.5 even though generating intermediate reasoning tokens considerably improves the results. Yet, models remain challenged especially on questions about study limitations and statistical analysis. CareMedEval provides a challenging benchmark for grounded reasoning, exposing current LLM limitations and paving the way for future development of automated support for critical appraisal.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 18 of its 36 references that are in the map. First posted on arXiv 5 Nov 2025.

Builds on

18 of 36 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024BioRAG: A RAG-LLM Framework for Biological Question Reasoning2 Aug

Other references · 17

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  5. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  6. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  7. 2023When "A Helpful Assistant" Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Modelscited by 50
  8. 2024Needle in the Haystack for Memory Based Large Language Modelscited by 32
  9. 2024BiomedRAG: A Retrieval Augmented Large Language Model for Biomedicinecited by 30
  10. 2023FrenchMedMCQA: A French Multiple-Choice Question Answering Dataset for Medical domaincited by 25
  11. 2024Biomedical Large Languages Models Seem not to be Superior to Generalist Models on Unseen Medical Datacited by 21
  12. 2023Appraising the Potential Uses and Harms of LLMs for Medical Systematic Reviewscited by 18
5 more
  1. 2024A Survey on Large Language Models from General Purpose to Medical Applications: Datasets, Methodologies, and Evaluationscited by 16
  2. 2024SciDQA: A Deep Reading Comprehension Dataset over Scientific Paperscited by 15
  3. 2024DrBenchmark: A Large Language Understanding Evaluation Benchmark for French Biomedical Domaincited by 11
  4. 2024Benchmarking Retrieval-Augmented Large Language Models in Biomedical NLP: Application, Robustness, and Self-Awarenesscited by 10
  5. 2024Adaptation of Biomedical and Clinical Pretrained Models to French Long Documents: A Comparative Studycited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Automating Expert-Level Medical Reasoning Evaluation of Large Language Models
  2. 2025Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases
  3. 2026Medical Reasoning with Large Language Models: A Survey and MR-Bench
  4. 2024A Review on Scientific Knowledge Extraction using Large Language Models in Biomedical Sciences
  5. 2025Introducing Answered with Evidence -- a framework for evaluating whether LLM responses to biomedical questions are founded in evidence
  6. 2026A primer on evaluation methods for large language models in healthcare
  7. 2024A Survey for Large Language Models in Biomedicine
  8. 2025Evaluating Large Language Models for Evidence-Based Clinical Question Answering