# BioNLI: Generating a Biomedical NLI Dataset Using Lexico-semantic Constraints for Adversarial Examples

arXiv: https://arxiv.org/abs/2210.14814 · HTML: https://mlsanity.com/paper/2210.14814/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 38 references that are in the map.
- First posted on arXiv 26 Oct 2022.

## Builds on (most cited first, 21 of 21)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022)
- [ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing](https://mlsanity.com/paper/1902.07669/) (2019)
- [Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models](https://mlsanity.com/paper/2101.00288/) (2021)
- [LinkBERT: Pretraining Language Models with Document Links](https://mlsanity.com/paper/2203.15827/) (2022)
- [NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics](https://mlsanity.com/paper/2112.08726/) (2021)
- [Lessons from Natural Language Inference in the Clinical Domain](https://mlsanity.com/paper/1808.06752/) (2018)
- [NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints](https://mlsanity.com/paper/2010.12884/) (2020)
- [DocNLI: A Large-scale Dataset for Document-level Natural Language Inference](https://mlsanity.com/paper/2106.09449/) (2021)
- [Controlled Text Generation as Continuous Optimization with Multiple Constraints](https://mlsanity.com/paper/2108.01850/) (2021)
- [Falsesum: Generating Document-level NLI Examples for Recognizing Factual Inconsistency in Summarization](https://mlsanity.com/paper/2205.06009/) (2022)
- [The RepEval 2017 Shared Task: Multi-Genre Natural Language Inference with Sentence Representations](https://mlsanity.com/paper/1707.08172/) (2017)
- [Faithfully Explainable Recommendation via Neural Logic Reasoning](https://mlsanity.com/paper/2104.07869/) (2021)
- [SuMe: A Dataset Towards Summarizing Biomedical Mechanisms](https://mlsanity.com/paper/2205.04652/) (2022)

## Built on by (most cited first, 15 of 15)

- [A Survey of Large Language Models in Medicine: Progress, Application, and Challenge](https://mlsanity.com/paper/2311.05112/) (2023)
- [Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluations](https://mlsanity.com/paper/2306.04618/) (2023)
- [Efficient Continual Pre-training by Mitigating the Stability Gap](https://mlsanity.com/paper/2406.14833/) (2024)
- [Large Language Model Benchmarks in Medical Tasks](https://mlsanity.com/paper/2410.21348/) (2024)
- [Benchmarking Retrieval-Augmented Large Language Models in Biomedical NLP: Application, Robustness, and Self-Awareness](https://mlsanity.com/paper/2405.08151/) (2024)
- [Me LLaMA: Foundation Large Language Models for Medical Applications](https://mlsanity.com/paper/2402.12749/) (2024)
- [If We May De-Presuppose: Robustly Verifying Claims through Presupposition-Free Question Decomposition](https://mlsanity.com/paper/2508.16838/) (2025)
- [BioMistral-NLU: Towards More Generalizable Medical Language Understanding through Instruction Tuning](https://mlsanity.com/paper/2410.18955/) (2024)
- [FinNLI: Novel Dataset for Multi-Genre Financial Natural Language Inference Benchmarking](https://mlsanity.com/paper/2504.16188/) (2025)
- [VerifAI: A Verifiable Open-Source Search Engine for Biomedical Question Answering](https://mlsanity.com/paper/2604.08549/) (2026)
- [Revealing Trends in Datasets from the 2022 ACL and EMNLP Conferences](https://mlsanity.com/paper/2404.08666/) (2024)
- [What Are Research Hypotheses?](https://mlsanity.com/paper/2509.00185/) (2025)
- [A Parameter-Efficient Transfer Learning Approach through Multitask Prompt Distillation and Decomposition for Clinical NLP](https://mlsanity.com/paper/2604.06650/) (2026)
- [Uncertainty-Aware Web-Conditioned Scientific Fact-Checking](https://mlsanity.com/paper/2604.11036/) (2026)
- [HealMed: Multilingual Evaluation of Large Language Models in Medicine](https://mlsanity.com/paper/2608.19981/) (2026)

## Abstract (arXiv)

Natural language inference (NLI) is critical for complex decision-making in biomedical domain. One key question, for example, is whether a given biomedical mechanism is supported by experimental evidence. This can be seen as an NLI problem but there are no directly usable datasets to address this. The main challenge is that manually creating informative negative examples for this task is difficult and expensive. We introduce a novel semi-supervised procedure that bootstraps an NLI dataset from existing biomedical dataset that pairs mechanisms with experimental evidence in abstracts. We generate a range of negative examples using nine strategies that manipulate the structure of the underlying mechanisms both with rules, e.g., flip the roles of the entities in the interaction, and, more importantly, as perturbations via logical constraints in a neuro-logical decoding system. We use this procedure to create a novel dataset for NLI in the biomedical domain, called BioNLI and benchmark two state-of-the-art biomedical classifiers. The best result we obtain is around mid 70s in F1, suggesting the difficulty of the task. Critically, the performance on the different classes of negative examples varies widely, from 97% F1 on the simple role change negative examples, to barely better than chance on the negative examples generated using neuro-logic decoding.