# Evaluating Paraphrastic Robustness in Textual Entailment Models

arXiv: https://arxiv.org/abs/2306.16722 · HTML: https://mlsanity.com/paper/2306.16722/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 57 references that are in the map.
- First posted on arXiv 29 Jun 2023.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [PAWS: Paraphrase Adversaries from Word Scrambling](https://mlsanity.com/paper/1904.01130/) (2019)
- [Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach](https://mlsanity.com/paper/1909.00161/) (2019)
- [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017)
- [Hypothesis Only Baselines in Natural Language Inference](https://mlsanity.com/paper/1805.01042/) (2018)
- [Adversarial Example Generation with Syntactically Controlled Paraphrase Networks](https://mlsanity.com/paper/1804.06059/) (2018)
- [Stress Test Evaluation for Natural Language Inference](https://mlsanity.com/paper/1806.00692/) (2018)
- [WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation](https://mlsanity.com/paper/2201.05955/) (2022)
- [Generating Natural Adversarial Examples](https://mlsanity.com/paper/1710.11342/) (2017)
- [Ask Me Anything: A simple strategy for prompting language models](https://mlsanity.com/paper/2210.02441/) (2022)
- [Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition](https://mlsanity.com/paper/2004.03066/) (2020)
- [Perturbation Sensitivity Analysis to Detect Unintended Model Biases](https://mlsanity.com/paper/1910.04210/) (2019)
- [Probing Natural Language Inference Models through Semantic Fragments](https://mlsanity.com/paper/1909.07521/) (2019)
- [Can neural networks understand monotonicity reasoning?](https://mlsanity.com/paper/1906.06448/) (2019)
- [CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation](https://mlsanity.com/paper/2211.00295/) (2022)
- [HELP: A Dataset for Identifying Shortcomings of Neural Models in Monotonicity Reasoning](https://mlsanity.com/paper/1904.12166/) (2019)
- [Falsesum: Generating Document-level NLI Examples for Recognizing Factual Inconsistency in Summarization](https://mlsanity.com/paper/2205.06009/) (2022)
- [Figurative Language in Recognizing Textual Entailment](https://mlsanity.com/paper/2106.01195/) (2021)
- [NOPE: A Corpus of Naturally-Occurring Presuppositions in English](https://mlsanity.com/paper/2109.06987/) (2021)
- [Exploring Transitivity in Neural NLI Models through Veridicality](https://mlsanity.com/paper/2101.10713/) (2021)

## Built on by (most cited first, 7 of 7)

- [Fine-Grained Natural Language Inference Based Faithfulness Evaluation for Diverse Summarisation Tasks](https://mlsanity.com/paper/2402.17630/) (2024)
- [NLI under the Microscope: What Atomic Hypothesis Decomposition Reveals](https://mlsanity.com/paper/2502.08080/) (2025)
- [MuSciClaims: Multimodal Scientific Claim Verification](https://mlsanity.com/paper/2506.04585/) (2025)
- [How often are errors in natural language reasoning due to paraphrastic variability?](https://mlsanity.com/paper/2404.11717/) (2024)
- [Same Patient, Different Words, Different Diagnosis? Evaluating Semantic Stability in Clinical LLMs](https://mlsanity.com/paper/2605.30646/) (2026)
- [Simpler becomes Harder: Do LLMs Exhibit a Coherent Behavior on Simplified Corpora?](https://mlsanity.com/paper/2404.06838/) (2024)
- [MERGE: Minimal Expression-Replacement GEneralization Test for Natural Language Inference](https://mlsanity.com/paper/2510.24295/) (2025)

## Abstract (arXiv)

We present PaRTE, a collection of 1,126 pairs of Recognizing Textual Entailment (RTE) examples to evaluate whether models are robust to paraphrasing. We posit that if RTE models understand language, their predictions should be consistent across inputs that share the same meaning. We use the evaluation set to determine if RTE models' predictions change when examples are paraphrased. In our experiments, contemporary models change their predictions on 8-16% of paraphrased examples, indicating that there is still room for improvement.