mlsanityData to 7 Oct 2026

SemEval-2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials

arXiv 2404.04963 ↗First posted 7 Apr 2024cs.CL cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are at the forefront of NLP achievements but fall short in dealing with shortcut learning, factual inconsistency, and vulnerability to adversarial inputs.These shortcomings are especially critical in medical contexts, where they can misrepresent actual model capabilities. Addressing this, we present SemEval-2024 Task 2: Safe Biomedical Natural Language Inference for ClinicalTrials. Our contributions include the refined NLI4CT-P dataset (i.e., Natural Language Inference for Clinical Trials - Perturbed), designed to challenge LLMs with interventional and causal reasoning tasks, along with a comprehensive evaluation of methods and results for participant submissions. A total of 106 participants registered for the task contributing to over 1200 individual submissions and 25 system overview papers. This initiative aims to advance the robustness and applicability of NLI models in healthcare, ensuring safer and more dependable AI assistance in clinical decision-making. We anticipate that the dataset, models, and outcomes of this task can support future research in the field of biomedical NLI. The dataset, competition leaderboard, and website are publicly available.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 30 of its 63 references that are in the map. First posted on arXiv 7 Apr 2024.

Builds on

30 of 63 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  4. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  5. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  6. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  7. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  8. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  9. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  10. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  11. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  12. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291
18 more
  1. 2018Lessons from Natural Language Inference in the Clinical Domaincited by 149
  2. 2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
  3. 2020The Effect of Natural Distribution Shift on Question Answering Modelscited by 104
  4. 2020BioMegatron: Larger Biomedical Domain Language Modelcited by 76
  5. 2022A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Modelscited by 68
  6. 2018Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailmentcited by 58
  7. 2019EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inferencecited by 39
  8. 2020Evidence Inference 2.0: More Data, Better Modelscited by 39
  9. 2020Tasty Burgers, Soggy Fries: Probing Aspect Robustness in Aspect-Based Sentiment Analysiscited by 23
  10. 2020Natural Language Inference in Context -- Investigating Contextual Reasoning over Long Textscited by 23
  11. 2023NLI4CT: Multi-Evidence Natural Language Inference for Clinical Trial Reportscited by 18
  12. 2019Meta-Learning surrogate models for sequential decision makingcited by 14
  13. 2019Deep Contextualized Biomedical Abbreviation Expansioncited by 14
  14. 2023Sebis at SemEval-2023 Task 7: A Joint System for Natural Language Inference and Evidence Retrieval from Clinical Trial Reportscited by 9
  15. 2023THiFLY Research at SemEval-2023 Task 7: A Multi-granularity System for CTR-based Textual Entailment and Evidence Retrievalcited by 4
  16. 2023Estimating the Causal Effects of Natural Logic Features in Neural NLI Modelscited by 3
  17. 2024Edinburgh Clinical NLP at SemEval-2024 Task 2: Fine-tune your model unless you have access to GPT-4cited by 2
  18. 2024SEME at SemEval-2024 Task 2: Comparing Masked and Generative Language Models on Natural Language Inference for Clinical Trialscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 8; 2026: 4.

20 papers.

Sort:

By area:

  1. 2025Medical Reasoning in the Era of LLMs: A Systematic Review of Enhancement Techniques and Applicationscited by 10
  2. 2024BioMistral-NLU: Towards More Generalizable Medical Language Understanding through Instruction Tuningcited by 4
  3. 2025BioMol-MQA: A Multi-Modal Question Answering Dataset For LLM Reasoning Over Bio-Molecular Interactionscited by 4
  4. 2025DeVisE: Behavioral Testing of Medical Large Language Modelscited by 2
  5. 2025The Knowledge-Reasoning Dissociation: Fundamental Limitations of LLMs in Clinical Natural Language Inferencecited by 2
  6. 2024DKE-Research at SemEval-2024 Task 2: Incorporating Data Augmentation with Generative Models and Biomedical Knowledge to Enhance Inference Robustnesscited by 1
  7. 2024FZI-WIM at SemEval-2024 Task 2: Self-Consistent CoT for Complex NLI in Biomedical Domaincited by 1
  8. 2025DeKeyNLU: Enhancing Natural Language to SQL Generation through Task Decomposition and Keyword Extractioncited by 1
  9. 2024TLDR at SemEval-2024 Task 2: T5-generated clinical-Language summaries for DeBERTa Report Analysiscited by 0
  10. 2024DFKI-NLP at SemEval-2024 Task 2: Towards Robust LLMs Using Data Perturbations and MinMax Trainingcited by 0
  11. 2024CRCL at SemEval-2024 Task 2: Simple prompt optimizationscited by 0
  12. 2024D-NLP at SemEval-2024 Task 2: Evaluating Clinical Inference Capabilities of Large Language Modelscited by 0
8 more
  1. 2024Lisbon Computational Linguists at SemEval-2024 Task 2: Using A Mistral 7B Model and Data Augmentationcited by 0
  2. 2025Am I eligible? Natural Language Inference for Clinical Trial Patient Recruitment: the Patient's Point of Viewcited by 0
  3. 2025A MISMATCHED Benchmark for Scientific Natural Language Inferencecited by 0
  4. 2025Compartmentalised Agentic Reasoning for Clinical NLIcited by 0
  5. 2026When Verification Fails: How Compositionally Infeasible Claims Escape Rejectioncited by 0
  6. 2026Neurosymbolic Clinical Trial Matching via LLM-Driven Abduction and Logical Verificationcited by 0
  7. 2026ANCHOR-RE: An Agentic Neuro-Symbolic Framework for Grounded Biomedical Relation Extractioncited by 0
  8. 2026UniBuc at SemEval-2024 Task 2: Tailored Prompting with Solar for Clinical NLIcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024IITK at SemEval-2024 Task 2: Exploring the Capabilities of LLMs for Safe Biomedical Natural Language Inference for Clinical Trials
  2. 2024A Survey for Large Language Models in Biomedicine
  3. 2024Biomedical Large Languages Models Seem not to be Superior to Generalist Models on Unseen Medical Data
  4. 2023Benchmarking large language models for biomedical natural language processing applications and recommendations
  5. 2025Disentangling Reasoning and Knowledge in Medical Large Language Models
  6. 2025Building a Human-Verified Clinical Reasoning Dataset via a Human LLM Hybrid Pipeline for Trustworthy Medical AI
  7. 2025RiskAgent: Synergizing Language Models with Validated Tools for Evidence-Based Risk Prediction
  8. 2023Evaluation of ChatGPT Family of Models for Biomedical Reasoning and Classification