# Edinburgh Clinical NLP at SemEval-2024 Task 2: Fine-tune your model unless you have access to GPT-4

arXiv: https://arxiv.org/abs/2404.00484 · HTML: https://mlsanity.com/paper/2404.00484/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 18 references that are in the map.
- First posted on arXiv 30 Mar 2024.

## Builds on (most cited first, 9 of 9)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Can large language models reason about medical questions?](https://mlsanity.com/paper/2207.08143/) (2022)
- [Do We Still Need Clinical Language Models?](https://mlsanity.com/paper/2302.08091/) (2023)
- [Parameter-Efficient Fine-Tuning of LLaMA for the Clinical Domain](https://mlsanity.com/paper/2307.03042/) (2023)
- [NLI4CT: Multi-Evidence Natural Language Inference for Clinical Trial Reports](https://mlsanity.com/paper/2305.03598/) (2023)
- [Can GPT-3.5 Generate and Code Discharge Summaries?](https://mlsanity.com/paper/2401.13512/) (2024)

## Built on by (most cited first, 2 of 2)

- [SemEval-2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials](https://mlsanity.com/paper/2404.04963/) (2024)
- [Edinburgh Clinical NLP at MEDIQA-CORR 2024: Guiding Large Language Models with Hints](https://mlsanity.com/paper/2405.18028/) (2024)

## Abstract (arXiv)

The NLI4CT task assesses Natural Language Inference systems in predicting whether hypotheses entail or contradict evidence from Clinical Trial Reports. In this study, we evaluate various Large Language Models (LLMs) with multiple strategies, including Chain-of-Thought, In-Context Learning, and Parameter-Efficient Fine-Tuning (PEFT). We propose a PEFT method to improve the consistency of LLMs by merging adapters that were fine-tuned separately using triplet and language modelling objectives. We found that merging the two PEFT adapters improves the F1 score (+0.0346) and consistency (+0.152) of the LLMs. However, our novel methods did not produce more accurate results than GPT-4 in terms of faithfulness and consistency. Averaging the three metrics, GPT-4 ranks joint-first in the competition with 0.8328. Finally, our contamination analysis with GPT-4 indicates that there was no test data leakage.