# A Course Shared Task on Evaluating LLM Output for Clinical Questions

arXiv: https://arxiv.org/abs/2408.00122 · HTML: https://mlsanity.com/paper/2408.00122/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 9 references that are in the map.
- First posted on arXiv 31 Jul 2024.

## Builds on (most cited first, 7 of 7)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Fact or Fiction: Verifying Scientific Claims](https://mlsanity.com/paper/2004.14974/) (2020)
- [Explainable Automated Fact-Checking for Public Health Claims](https://mlsanity.com/paper/2010.09926/) (2020)
- [Missing Counter-Evidence Renders NLP Fact-Checking Unrealistic for Misinformation](https://mlsanity.com/paper/2210.13865/) (2022)
- [WikiContradict: A Benchmark for Evaluating LLMs on Real-World Knowledge Conflicts from Wikipedia](https://mlsanity.com/paper/2406.13805/) (2024)
- [Missci: Reconstructing Fallacies in Misrepresented Science](https://mlsanity.com/paper/2406.03181/) (2024)

## Abstract (arXiv)

This paper presents a shared task that we organized at the Foundations of Language Technology (FoLT) course in 2023/2024 at the Technical University of Darmstadt, which focuses on evaluating the output of Large Language Models (LLMs) in generating harmful answers to health-related clinical questions. We describe the task design considerations and report the feedback we received from the students. We expect the task and the findings reported in this paper to be relevant for instructors teaching natural language processing (NLP) and designing course assignments.